Database sharding method, apparatus, device, and computer-readable storage medium

Abstract

The application relates to the field of financial technology data processing, and discloses a database fragmentation method, which comprises the following steps: extracting operation objects and operation conditions of a historical data operation instruction set of a target database group, calculating the coincidence degrees between each operation object, taking operation objects with a coincidence degree greater than a preset coincidence threshold as the same fragmentation object, screening core operation conditions from the operation conditions corresponding to the same fragmentation object as fragmentation conditions, calculating the fragmentation size of the fragmentation object according to the actual value corresponding to the fragmentation condition and the value range, fragmenting the corresponding fragmentation object according to the fragmentation condition and the fragmentation size, and sequentially publishing the fragmentation to corresponding servers according to the maximum load value of the servers corresponding to the target database group. The application further discloses a database fragmentation device, an electronic device and a computer readable storage medium. The application can improve the management efficiency of business data of an insurance business system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of financial technology data processing, and in particular to a database sharding method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] It can be understood that the types of insurance business are very diverse, such as property insurance, life insurance, accident insurance, unemployment insurance, etc. Corresponding operations such as insurance, claims, policy changes or cancellation will involve large amounts of data operations, so the insurance business data system must have the ability to support high-concurrency storage and access of data.

[0003] Generally, the insurance business data system realizes the storage of business data by month table, year table or day table. When a business data involves adding, updating, deleting or querying operations, it needs to traverse the month table, year table or day table involved in the business data, which reduces the data query efficiency, and the business data is not easy to maintain and migrate, and the overall business data management efficiency is low. SUMMARY

[0004] The present application provides a database sharding method, device, electronic equipment and computer readable storage medium, which mainly aims to improve the management efficiency of business data of the insurance business system.

[0005] To achieve the above purpose, the present application provides a database sharding method, which comprises:

[0006] Obtain a set of historical data operation instructions of a target database group, extract the operation object and operation condition corresponding to each data operation instruction in the set of historical data operation instructions;

[0007] Calculate the coincidence degree between each operation object, and take the operation object with a coincidence degree greater than a preset coincidence threshold as a same shard object;

[0008] Filter the core operation condition from the operation condition corresponding to the same shard object as a shard condition;

[0009] Calculate the shard size of the shard object according to the actual value and value range corresponding to the shard condition;

[0010] Shard the shard object according to the shard condition and corresponding shard size, and collect all shards of the target database group to obtain a shard set;

[0011] Obtain a server set corresponding to the target database group and the maximum load value of each server in the server set, and sequentially publish the shards in the shard set to the corresponding servers according to the maximum load value.

[0012] Optionally, the extracting the operation object and the operation condition corresponding to each data operation instruction in the historical data operation instruction set comprises:

[0013] According to a preset instruction syntax, each data operation instruction is sequentially labeled to obtain a to-be-operated table, a to-be-operated field and a condition field;

[0014] The to-be-operated table and the to-be-operated field are taken as the operation object;

[0015] The condition field is taken as the operation condition.

[0016] Optionally, the calculating the coincidence degree between each operation object comprises:

[0017] According to a preset operation table weight and a preset operation field weight, the to-be-operated table and the to-be-operated field of each operation object are weighted;

[0018] An operation object is sequentially selected from all operation objects as a reference object, and other operation objects are taken as comparison objects;

[0019] According to the principle that the weight of the to-be-operated table is added only when the to-be-operated tables are the same, and the weight of the to-be-operated field is added only when the to-be-operated fields are the same, the sum of the weights between the reference object and any comparison object is calculated as the coincidence degree between the reference object and the corresponding comparison object.

[0020] Optionally, the filtering a core operation condition as a sharding condition from the operation condition corresponding to the same sharding object comprises:

[0021] It is judged whether the condition field corresponding to the operation condition in each data operation instruction of the same sharding object is unique;

[0022] When the condition field is unique, the weight of the corresponding condition field is a preset unit value;

[0023] When the condition field is not unique, the weight of each condition field is a ratio of the preset unit value to the number of condition fields;

[0024] The weights of the same condition field in all data operation instructions of the same sharding object are summed to obtain a total weight of each condition field;

[0025] The condition field with a weight total value greater than a preset weight threshold is selected as a core operation condition to obtain the sharding condition.

[0026] Optionally, the calculating the sharding size of the sharding object according to the actual value and the value range corresponding to the sharding condition comprises:

[0027] Convert the value range of the sharding condition into a hash data space;

[0028] Publishing all actual values ​​corresponding to the sharding condition into the hash data space, and determining whether the actual values ​​of the sharding condition are evenly distributed in the hash data space;

[0029] When the actual values ​​of the sharding condition are evenly distributed, the sharding size of the corresponding sharding object is set to the default sharding size;

[0030] When the actual values ​​of the sharding condition are unevenly distributed, sharding critical points are divided according to the distribution of the actual values ​​of the sharding condition, and the sharding size of the corresponding sharding object is calculated according to the sharding critical points.

[0031] Optionally, publishing the shards in the shard set to corresponding servers in sequence according to the maximum load value includes:

[0032] Delivering one shard in the shard set to a server in the server set in sequence, and calculating the load difference between the actual load of each server and the corresponding maximum load value;

[0033] When the load difference is less than a preset difference warning value, returning to the above step of sequentially delivering a shard in the shard set to a server in the server set;

[0034] When the load difference is not less than the preset difference warning value, the corresponding server is removed from the server set.

[0035] In order to solve the above problems, the present invention further provides a database sharding device, the device comprising:

[0036] A data operation instruction parsing module is used to obtain a historical data operation instruction set of a target database group and extract the operation object and operation condition corresponding to each data operation instruction in the historical data operation instruction set;

[0037] A sharding strategy generation module is configured to calculate the overlap between each of the operation objects, treat the operation objects with an overlap greater than a preset overlap threshold as the same sharding object, select core operation conditions from the operation conditions corresponding to the same sharding object as sharding conditions, and calculate the sharding size of the sharding object based on the actual values ​​and value ranges corresponding to the sharding conditions;

[0038] The sharding division and deployment module is used to shard the shard object according to the sharding conditions and the corresponding shard sizes, collect all the shards of the target database group to obtain a shard set, obtain the server set corresponding to the target database group, and the maximum load value of each server in the server set, and publish the shards in the shard set to the corresponding servers in sequence according to the maximum load value.

[0039] Optionally, the slice object identification module calculates the overlap between each of the operation objects by the following method:

[0040] Assigning weights to the table to be operated and the field to be operated of each operation object according to the preset operation table weights and the preset operation field weights;

[0041] Select one operation object from all operation objects in turn as a reference object, and the other operation objects as comparison objects;

[0042] According to the principle that if the tables to be operated are the same, the weights of the tables to be operated are added together, and if the fields to be operated are the same, the weights of the fields to be operated are added together, the sum of the weights between the reference object and any comparison object is calculated as the degree of overlap between the reference object and the corresponding comparison object.

[0043] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0044] a memory storing at least one computer program; and

[0045] The processor executes the program stored in the memory to implement the above-mentioned database sharding method.

[0046] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned database sharding method.

[0047] The present invention extracts the operation objects and operation conditions from the historical data operation instruction set of the target database group corresponding to the insurance business system, determines the sharding objects of the target database group according to the operation objects, and determines the sharding conditions and sharding size of the sharding objects according to the operation objects, so that the final sharding result conforms to the distribution of the historical operation data of the insurance business system, thereby facilitating the improvement of the management efficiency of querying, modifying, adding, etc. of the data of the insurance business system. Furthermore, in combination with the load conditions of the server corresponding to the target database group, the shards are reasonably distributed to different servers, which is conducive to the rational use of the server resources of the insurance business system and improves the data management efficiency of the insurance business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a process for sharding a database according to an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a detailed implementation flow of one step of a database sharding method provided in one embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a detailed implementation flow of one step of a database sharding method provided in one embodiment of the present invention;

[0051] Figure 4 A schematic diagram of a detailed implementation flow of one step of a database sharding method provided in one embodiment of the present invention;

[0052] Figure 5 A functional module diagram of a database sharding device provided by one embodiment of the present invention;

[0053] Figure 6 A schematic diagram of the structure of an electronic device for implementing the database sharding method provided in one embodiment of the present invention.

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] The embodiment of the present application provides a database sharding method. The execution subject of the database sharding method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the database sharding method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0057] Reference Figure 1 FIG. 1 is a flow chart of a database sharding method according to an embodiment of the present invention. In this embodiment, the database sharding method includes:

[0058] S1. Obtain a historical data operation instruction set of a target database group, and extract the operation object and operation condition corresponding to each data operation instruction in the historical data operation instruction set;

[0059] This embodiment of the present invention uses the financial insurance sector as an example to illustrate a database sharding method. Specifically, the target database group refers to multiple databases corresponding to a pre-defined insurance business system, which are used to store insurance-related data, such as user insurance applications, claims, and insurance changes.

[0060] Preferably, the target database group has completed database and table sharding operations based on relevant attributes of the insurance business. For example, database and table sharding is performed based on the type of insurance business, ensuring that each database in the target database group stores insurance business data of different business types, such as an auto insurance business database, a life insurance business database, an accident insurance business database, an unemployment insurance business database, etc.

[0061] In an embodiment of the present invention, the historical data operation instruction set is a set of instructions for querying, modifying, adding, deleting, and other operations on the data in the target database group within a preset database shard observation period, which can be two months, a quarter, or half a year.

[0062] In an embodiment of the present invention, the historical data operation instruction set is used as a reference for database sharding, with the aim of ensuring that the final database sharding result conforms to the data operation tendency reflected by the historical data operation instruction set, for example, the distribution of operation objects corresponding to the historical data operation instruction set and the distribution of operation conditions for the same operation object.

[0063] Specifically, extracting the operation object and operation condition corresponding to each data operation instruction in the historical data operation instruction set includes:

[0064] According to the preset instruction syntax, each of the data operation instructions is marked in turn to obtain a table to be operated, a field to be operated, and a condition field;

[0065] Taking the table to be operated and the field to be operated as the operation object;

[0066] The condition field is used as the operation condition.

[0067] In an embodiment of the present invention, the preset instruction syntax refers to the grammatical knowledge corresponding to the database operation language supported by the target database group. For example, if a database uses SQL (structured query language) for data maintenance, then the preset instruction syntax is SQL syntax, and the SQL syntax is used to identify the table to be operated, the field to be operated, and the specific condition field in a data operation instruction.

[0068] S2. Calculate the overlap between each of the operation objects, and treat the operation objects with an overlap greater than a preset overlap threshold as the same shard object;

[0069] It can be understood that, assuming that the historical data operation instruction set contains tens of thousands of instructions, it will correspondingly involve tens of thousands of operation objects consisting of tables to be operated or fields to be operated. The operation objects have a certain degree of repetitiveness. By calculating the degree of overlap between each of the operation objects, the operation objects corresponding to the historical data operation instruction set can be deduplicated.

[0070] In this embodiment of the present invention, by treating operation objects with a degree of overlap greater than a preset overlap threshold as the same sharding object, the target object of database sharding can be guaranteed to be consistent with the historical data operation object, which is beneficial for improving the efficiency of subsequent database data operations. For example, if the sharding object of a life insurance business database is the user ID of a certain brand of life insurance, the life insurance business database can be sharded based on the information of this user ID.

[0071] For details, see Figure 2 As shown, the calculation of the overlap between each of the operation objects includes:

[0072] S21. Assigning weights to the to-be-operated table and to-be-operated field of each operation object according to a preset operation table weight and a preset operation field weight;

[0073] S22, selecting one operation object from all operation objects in turn as a reference object, and the other operation objects as comparison objects;

[0074] S23. Based on the principle that if the tables to be operated are the same, the weights of the tables to be operated are added together; if the fields to be operated are the same, the weights of the fields to be operated are added together together. Calculate the sum of the weights between the reference object and any comparison object as the degree of overlap between the reference object and the corresponding comparison object.

[0075] In the embodiment of the present application, considering that the same operation field can exist in different operation tables, for example, a user with user ID 03590 purchases A brand life insurance and also purchases B brand life insurance, the corresponding user ID field exists in the A brand insurance business details table and also exists in the B brand insurance business details table. Therefore, the preset operation table weight can be set to be higher than the preset operation field weight, for example, the preset operation table weight is 1 and the preset operation field is 0.5.

[0076] Exemplarily, the historical data operation instruction set includes the following instructions, the preset operation table weight is 1 and the preset operation field is 0.5.

[0077] Historical data operation instruction 1: the table to be operated is the A brand life insurance business table and the field to be operated is the user ID.

[0078] Historical data operation instruction 2: the table to be operated is the B brand life insurance business table and the field to be operated is the insurance premium unit price.

[0079] Historical data operation instruction 3: the table to be operated is the A brand life insurance business table and the field to be operated is the user ID.

[0080] Historical data operation instruction 4: the table to be operated is the B brand life insurance business table and the field to be operated is the user ID.

[0081] Correspondingly, the coincidence degree between instruction 1 and instruction 2 is 0, the coincidence degree between instruction 1 and instruction 3 is 3, and the coincidence degree between instruction 1 and instruction 4 is 1; the coincidence degree between instruction 2 and instruction 3 is 0, the coincidence degree between instruction 2 and instruction 4 is 2, and the coincidence degree between instruction 3 and instruction 4 is 1.

[0082] In the embodiment of the present application, the preset coincidence threshold can be set according to actual business conditions.

[0083] S3, filtering the core operation condition as a sharding condition from the operation condition corresponding to the same sharding object, calculating the sharding size of the sharding object according to the actual value and the value range corresponding to the sharding condition;

[0084] It can be understood that after the sharding object is determined, it is further necessary to consider how to shard the sharding object and the size of each shard, for example, assuming that the sharding object is the user ID of the A brand, whether the user ID of the A brand is sharded by using the user ID modulo method or the sharding object is sharded according to the ID value interval.

[0085] In detail, refer to Figure 3As shown, the core operation condition is screened from the operation condition corresponding to the same sharding object as the sharding condition, comprising:

[0086] S31, judging whether the condition field corresponding to the operation condition in each data operation instruction of the same sharding object is unique;

[0087] When the condition field is unique, then S32 is executed, the weight of the corresponding condition field is a preset unit value;

[0088] When the condition field is not unique, then S33 is executed, the weight of each condition field is the ratio of the preset unit value and the number of condition fields;

[0089] S34, summing the weight of the same condition field in all data operation instructions of the same sharding object to obtain the total weight of each condition field;

[0090] S35, selecting the condition field with a weight greater than a preset weight threshold as the core operation condition to obtain the sharding condition.

[0091] It can be understood that for the case that there are multiple operation conditions in the historical data operation instruction, for example, the user ID of the user who inquires about the A brand of life insurance and whose insurance time is more than 5 years and is female, the corresponding data operation instruction operation condition includes two conditions that the insurance time is greater than 5 years and the user gender is female. Assuming that the preset unit is 1, the weight of the condition field corresponding to the insurance time greater than 5 years is 0.5, and the weight of the condition field corresponding to the user gender is also 0.5.

[0092] In the embodiment of the application, the preset weight threshold can be set according to the actual business situation.

[0093] In the embodiment of the application, after determining the sharding condition by the above operation, the final sharding size needs to be further determined in combination with the actual value and the value range corresponding to the sharding condition.

[0094] In detail, referring to Figure 4 As shown, the sharding size of the sharding object is calculated according to the actual value and the value range corresponding to the sharding condition, comprising:

[0095] S311, converting the value range of the sharding condition into a hash data space;

[0096] S312, publishing all actual values corresponding to the sharding condition into the hash data space;

[0097] S313, judging whether the actual values of the sharding condition are uniformly distributed in the hash data space;

[0098] When the actual values of the sharding condition are uniformly distributed, then S314 is executed, and the sharding size of the corresponding sharding object is set as a default sharding size;

[0099] When the actual values of the sharding condition are non-uniformly distributed, then S315 is executed, the sharding critical point is divided according to the distribution of the actual values of the sharding condition, and the sharding size of the corresponding sharding object is calculated according to the sharding critical point.

[0100] In the embodiment of the application, the sharding object of the target database group is determined according to the operation object, and the sharding condition and the sharding size of the sharding object are determined according to the operation object, so that the final sharding result conforms to the distribution of the historical operation data of the insurance business system

[0101] S4, the sharding object is sharded according to the sharding condition and the corresponding sharding size, all sharding of the target database group is collected to obtain a sharding set, a server set corresponding to the target database group is obtained, and the maximum load value of each server in the server set is obtained, and the sharding in the sharding set is sequentially published to the corresponding server according to the maximum load value.

[0102] In the embodiment of the application, after the sharding of the target data group is completed, further, the sharding is reasonably published to different servers in combination with the load condition of the server corresponding to the target database group, which is beneficial to reasonably utilize the server resources of the insurance business system and improves the data management efficiency of the insurance business system.

[0103] In detail, the sharding in the sharding set is sequentially published to the corresponding server according to the maximum load value, including:

[0104] one sharding in the sharding set is sequentially put into one server in the server set, and the load difference value between the actual load of each server and the corresponding maximum load value is calculated;

[0105] When the load difference value is less than a preset difference warning value, the step of sequentially putting one sharding in the sharding set into one server in the server set is returned;

[0106] When the load difference value is not less than the preset difference warning value, the corresponding server is excluded from the server set.

[0107] In the embodiment of the application, the preset difference warning value can be set according to the actual situation.

[0108] In the embodiment of the present application, the load difference between the actual load of the server and the corresponding maximum load value is calculated each time the shard is put in the server, so that the resources of the server can be reasonably utilized and the load balancing of the server can be ensured.

[0109] The application extracts the operation object and operation condition in the historical data operation instruction set of the target database group corresponding to the insurance business system, determines the shard object of the target database group according to the operation object, and determines the shard condition and shard size of the shard object according to the operation object, so that the final shard result conforms to the distribution of the historical operation data of the insurance business system, thereby facilitating the improvement of the management efficiency of the query, modification, addition and the like of the data of the insurance business system, and further, in combination with the load condition of the server corresponding to the target database group, the shard is reasonably distributed to different servers, which is beneficial to the reasonable utilization of the server resources of the insurance business system and the improvement of the data management efficiency of the insurance business system.

[0110] As Figure 5 shown is a functional module diagram of a database sharding device provided by an embodiment of the present application.

[0111] The database sharding device 100 can be installed in an electronic device. According to the implemented functions, the database sharding device 100 includes a data operation instruction analysis module 101, a shard strategy generation module 102 and a shard division and deployment module 103. The modules of the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and are stored in the memory of the electronic device.

[0112] In the embodiment, the functions of each module / unit are as follows:

[0113] The data operation instruction analysis module 101 is configured to obtain a historical data operation instruction set of a target database group, and extract an operation object and an operation condition corresponding to each data operation instruction in the historical data operation instruction set;

[0114] The shard strategy generation module 102 is configured to calculate the coincidence degree between each operation object, take the operation objects with a coincidence degree greater than a preset coincidence threshold as the same shard object, filter core operation conditions from the operation conditions corresponding to the same shard object as shard conditions, and calculate the shard size of the shard object according to the actual value and the value range corresponding to the shard conditions.

[0115] The shard division and deployment module 103 is used to shard the shard object according to the shard conditions and the corresponding shard sizes, collect all the shards of the target database group to obtain a shard set, obtain the server set corresponding to the target database group, and the maximum load value of each server in the server set, and publish the shards in the shard set to the corresponding servers in sequence according to the maximum load value.

[0116] In detail, the specific implementation of each module of the database sharding device 100 is as follows:

[0117] Step 1: Obtain a historical data operation instruction set of the target database group, and extract the operation object and operation condition corresponding to each data operation instruction in the historical data operation instruction set;

[0118] This embodiment of the present invention uses the financial insurance sector as an example to illustrate a database sharding method. Specifically, the target database group refers to multiple databases corresponding to a pre-defined insurance business system, which are used to store insurance-related data, such as user insurance applications, claims, and insurance changes.

[0119] Preferably, the target database group has completed database and table sharding operations based on relevant attributes of the insurance business. For example, database and table sharding is performed based on the type of insurance business, ensuring that each database in the target database group stores insurance business data of different business types, such as an auto insurance business database, a life insurance business database, an accident insurance business database, an unemployment insurance business database, etc.

[0120] In an embodiment of the present invention, the historical data operation instruction set is a set of instructions for querying, modifying, adding, deleting, and other operations on the data in the target database group within a preset database shard observation period, which can be two months, a quarter, or half a year.

[0121] In an embodiment of the present invention, the historical data operation instruction set is used as a reference for database sharding, with the aim of ensuring that the final database sharding result conforms to the data operation tendency reflected by the historical data operation instruction set, for example, the distribution of operation objects corresponding to the historical data operation instruction set and the distribution of operation conditions for the same operation object.

[0122] Specifically, extracting the operation object and operation condition corresponding to each data operation instruction in the historical data operation instruction set includes:

[0123] According to the preset instruction syntax, each of the data operation instructions is marked in turn to obtain a table to be operated, a field to be operated, and a condition field;

[0124] Taking the table to be operated and the field to be operated as the operation object;

[0125] The condition field is used as the operation condition.

[0126] In an embodiment of the present invention, the preset instruction syntax refers to the grammatical knowledge corresponding to the database operation language supported by the target database group. For example, if a database uses SQL (structured query language) for data maintenance, then the preset instruction syntax is SQL syntax, and the SQL syntax is used to identify the table to be operated, the field to be operated, and the specific condition field in a data operation instruction.

[0127] Step 2: Calculate the overlap between each of the operation objects, and treat the operation objects whose overlap is greater than a preset overlap threshold as the same shard object;

[0128] It can be understood that, assuming that the historical data operation instruction set contains tens of thousands of instructions, it will correspondingly involve tens of thousands of operation objects consisting of tables to be operated or fields to be operated. The operation objects have a certain degree of repetitiveness. By calculating the degree of overlap between each of the operation objects, the operation objects corresponding to the historical data operation instruction set can be deduplicated.

[0129] In this embodiment of the present invention, by treating operation objects with a degree of overlap greater than a preset overlap threshold as the same sharding object, the target object of database sharding can be guaranteed to be consistent with the historical data operation object, which is beneficial for improving the efficiency of subsequent database data operations. For example, if the sharding object of a life insurance business database is the user ID of a certain brand of life insurance, the life insurance business database can be sharded based on the information of this user ID.

[0130] Specifically, the calculation of the overlap between each of the operation objects includes:

[0131] Assigning weights to the table to be operated and the field to be operated of each operation object according to the preset operation table weights and the preset operation field weights;

[0132] Select one operation object from all operation objects in turn as a reference object, and the other operation objects as comparison objects;

[0133] According to the principle that if the tables to be operated are the same, the weights of the tables to be operated are added together, and if the fields to be operated are the same, the weights of the fields to be operated are added together, the sum of the weights between the reference object and any comparison object is calculated as the degree of overlap between the reference object and the corresponding comparison object.

[0134] In an embodiment of the present invention, considering that the same operation field may exist in different operation tables, for example, if a user with user ID 03590 purchases life insurance from both Brand A and Brand B, the corresponding user ID field will exist in both the insurance business details table for Brand A and the insurance business details table for Brand B. Therefore, the preset operation table weight can be set to be higher than the preset operation field weight, for example, the preset operation table weight is 1, and the preset operation field weight is 0.5.

[0135] Exemplarily, the historical data operation instruction set includes the following instructions, the preset operation table weight is 1, and the preset operation field is 0.5:

[0136] Historical data operation instruction 1: The table to be operated is the Brand A life insurance business table, and the field to be operated is the user ID;

[0137] Historical data operation instruction 2: The table to be operated is the Brand B life insurance business table, and the field to be operated is the insured amount unit price;

[0138] Historical data operation instruction 3: The table to be operated is the Brand A life insurance business table, and the field to be operated is the user ID;

[0139] Historical data operation instruction 4: The table to be operated is the Brand B life insurance business table, and the field to be operated is the user ID;

[0140] Correspondingly, the overlap between instruction 1 and instruction 2 is 0, the overlap between instruction 1 and instruction 3 is 3, and the overlap between instruction 1 and instruction 4 is 1; the overlap between instruction 2 and instruction 3 is 0, the overlap between instruction 2 and instruction 4 is 2, and the overlap between instruction 3 and instruction 4 is 1.

[0141] In the embodiment of the present invention, the preset overlap threshold can be set according to actual business conditions.

[0142] Step 3: Select the core operation conditions from the operation conditions corresponding to the same shard object as the shard conditions, and calculate the shard size of the shard object according to the actual value and value range corresponding to the shard conditions;

[0143] It is understandable that after the sharding object is clear, it is necessary to further consider how to shard the sharding object and the size of each shard. For example, assuming that the sharding object is the user ID of brand A, for the user ID of brand A, should the sharding be performed by taking the modulo of the user ID, or should the sharding object be sharded according to the ID value range?

[0144] In detail, the step of selecting the core operation condition as the sharding condition from the operation conditions corresponding to the same sharding object includes:

[0145] determine whether the condition field corresponding to the operation condition in each data operation instruction of the same shard object is unique;

[0146] When the condition field is unique, the weight of the corresponding condition field is a preset unit value;

[0147] When the condition field is not unique, the weight of each condition field is the ratio of the preset unit value to the number of condition fields;

[0148] Sum the weights of the same condition field in all data operation instructions of the same shard object to obtain the total weight of each condition field;

[0149] Select the condition field with a weight greater than a preset weight threshold as the core operation condition to obtain the shard condition.

[0150] It can be understood that for the case where multiple operation conditions exist in historical data operation instructions, for example, the user ID of the person who inquires about A brand life insurance and whose insurance time is more than 5 years and is female, the corresponding data operation instruction operation condition includes two conditions of insurance time greater than 5 years and user gender female. Assuming that the preset unit is 1, the weight of the condition field corresponding to the insurance time greater than 5 years is 0.5, and the weight of the condition field corresponding to the user gender female is also 0.5.

[0151] In the embodiment of the application, the preset weight threshold can be set according to actual business conditions.

[0152] In the embodiment of the application, after determining the shard condition by the above operation, the final shard size needs to be further determined in combination with the actual value and the value range corresponding to the shard condition.

[0153] In detail, the calculation of the shard size of the shard object according to the actual value and the value range corresponding to the shard condition includes:

[0154] Converting the value range of the shard condition into a hash data space;

[0155] Publishing all actual values corresponding to the shard condition into the hash data space, and determining whether the actual values of the shard condition are uniformly distributed in the hash data space;

[0156] When the actual values of the shard condition are uniformly distributed, the shard size of the corresponding shard object is set to a default shard size;

[0157] When the actual value of the sharding condition is unevenly distributed, then the sharding critical point is divided according to the distribution of the actual value of the sharding condition, and the sharding size of the corresponding sharding object is calculated according to the sharding critical point.

[0158] In the embodiment of the application, the sharding object of the target database group is determined according to the operation object, and the sharding condition and the sharding size of the sharding object are determined according to the operation object, so that the final sharding result conforms to the distribution of the historical operation data of the insurance business system

[0159] Step four, the sharding object is sharded according to the sharding condition and the corresponding sharding size, all sharding of the target database group is collected to obtain a sharding set, a server set corresponding to the target database group is obtained, and the maximum load value of each server in the server set is obtained, and the sharding in the sharding set is sequentially published to the corresponding server according to the maximum load value.

[0160] In the embodiment of the application, after the sharding of the target data group is completed, further, the sharding is reasonably published to different servers in combination with the load condition of the server corresponding to the target database group, which is beneficial to reasonably utilize the server resources of the insurance business system and improves the data management efficiency of the insurance business system.

[0161] In detail, the sharding in the sharding set is sequentially published to the corresponding server according to the maximum load value, including:

[0162] one sharding in the sharding set is sequentially put into one server in the server set, and the load difference value between the actual load of each server and the corresponding maximum load value is calculated;

[0163] When the load difference value is less than a preset difference warning value, the step of sequentially putting one sharding in the sharding set into one server in the server set is returned;

[0164] When the load difference value is not less than the preset difference warning value, the corresponding server is excluded from the server set.

[0165] In the embodiment of the application, the preset difference warning value can be set according to the actual situation.

[0166] In the embodiment of the application, the load difference value between the actual load of the server and the corresponding maximum load value is calculated after the sharding is put into the server each time, and the operation can reasonably utilize the resources of the server and guarantee the load balance of the server.

[0167] The present invention extracts the operation objects and operation conditions from the historical data operation instruction set of the target database group corresponding to the insurance business system, determines the sharding objects of the target database group according to the operation objects, and determines the sharding conditions and sharding size of the sharding objects according to the operation objects, so that the final sharding result conforms to the distribution of the historical operation data of the insurance business system, thereby facilitating the improvement of the management efficiency of querying, modifying, adding, etc. of the data of the insurance business system. Furthermore, in combination with the load conditions of the server corresponding to the target database group, the shards are reasonably distributed to different servers, which is conducive to the rational use of the server resources of the insurance business system and improves the data management efficiency of the insurance business system.

[0168] like Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a database sharding method provided by an embodiment of the present invention.

[0169] The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a database shard.

[0170] The memory 11 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 1. Furthermore, the memory 11 may include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can be used not only to store application software installed in the electronic device 1 and various types of data, such as the code of database shards, but also to temporarily store data that has been output or is about to be output.

[0171] The processor 10 can be composed of integrated circuits in some embodiments, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, which connects various components of the entire electronic device through various interfaces and lines, and executes various functions of the electronic device 1 and processes data by running or executing programs or modules (such as database shards, etc.) stored in the memory 11 and calling data stored in the memory 11.

[0172] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 11 and the at least one processor 10, etc.

[0173] Figure 5 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0174] For example, although not shown, the electronic device 1 can also include a power supply (such as a battery) for powering various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so as to realize functions such as charge management, discharge management, and power consumption management through the power management device. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, etc. The electronic device 1 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0175] Further, the electronic device 1 can also include a network interface, which can optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually used to establish a communication connection between the electronic device 1 and other electronic devices.

[0176] Optionally, the electronic device 1 can further include a user interface, which can be a display, an input unit such as a keyboard, and optionally a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display can also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

[0177] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by the structure.

[0178] The database shard stored in the memory 11 in the electronic device 1 is a combination of a plurality of instructions, which, when running in the processor 10, can achieve:

[0179] Obtain a set of historical data operation instructions of a target database group, and extract an operation object and an operation condition corresponding to each data operation instruction in the set of historical data operation instructions;

[0180] Calculate the coincidence degree between each operation object, and take an operation object with a coincidence degree greater than a preset coincidence threshold as a same shard object;

[0181] Select a core operation condition from the operation conditions corresponding to the same shard object as a shard condition;

[0182] Calculate a shard size of the shard object according to an actual value corresponding to the shard condition and a value range;

[0183] Shard the shard object according to the shard condition and the corresponding shard size, and collect all shards of the target database group to obtain a shard set;

[0184] Obtain a server set corresponding to the target database group and a maximum load value of each server in the server set, and sequentially publish the shards in the shard set to the corresponding servers according to the maximum load value.

[0185] Further, the modules / units integrated in the electronic device 1 are stored in a computer readable storage medium if they are realized in the form of software function units and sold or used as independent products. The computer readable storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0186] The application further provides a computer readable storage medium, which stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device:

[0187] obtain a set of historical data operation instructions of a target database group, and extract operation objects and operation conditions corresponding to each data operation instruction in the set of historical data operation instructions;

[0188] calculate coincidence degrees between each operation object, and take operation objects with coincidence degrees greater than a preset coincidence threshold as the same shard object;

[0189] select core operation conditions from operation conditions corresponding to the same shard object as shard conditions;

[0190] calculate a shard size of the shard object according to an actual value corresponding to the shard condition and a value range;

[0191] perform sharding on the shard object according to the shard condition and the corresponding shard size, collect all shards of the target database group, and obtain a shard set;

[0192] obtain a server set corresponding to the target database group and a maximum load value of each server in the server set, and sequentially publish shards in the shard set to corresponding servers according to the maximum load value.

[0193] In addition, each function module in each embodiment of the application can be integrated in one processing unit, or each unit can be physically independent, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software function module.

[0194] It is obvious for those skilled in the art that the application is not limited to the details of the above exemplary embodiments, and the application can be realized in other specific forms without departing from the spirit or essential characteristics of the application.

[0195] Thus, the embodiments can be regarded as illustrative rather than restrictive, and the scope of the application is defined by the appended claims rather than the description preceding it, so that all changes that come within the meaning and range of equivalency of the claims are to be embraced within the scope of the application. Any reference signs in the claims should not be construed as limiting the scope of the claims.

[0196] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. The blockchain is essentially a decentralized database, and is a series of data blocks associated using cryptographic methods. Each data block contains information of a batch of network transactions, and is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0197] Embodiments of the present application can acquire and process related data based on holographic projection technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0198] In addition, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. The second and subsequent words are used to indicate the name, not any particular order.

[0199] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A database sharding method, characterized in that: The method comprises: Obtaining a historical data operation instruction set of a target database group, and extracting an operation object and an operation condition corresponding to each data operation instruction in the historical data operation instruction set; Calculating the overlap between each of the operation objects, and treating the operation objects with an overlap greater than a preset overlap threshold as the same shard object; Filter the core operation conditions from the operation conditions corresponding to the same shard object as the shard conditions; Calculate the shard size of the shard object according to the actual value and value range corresponding to the shard condition; Sharding the shard object according to the shard condition and the corresponding shard size, and gathering all shards of the target database group to obtain a shard set; Obtaining a server set corresponding to the target database group and a maximum load value of each server in the server set, and publishing the shards in the shard set to the corresponding servers in sequence according to the maximum load value; The calculating the fragment size of the fragment object according to the actual value and the value range corresponding to the fragment condition includes: Convert the value range of the sharding condition into a hash data space; Publishing all actual values ​​corresponding to the sharding condition into the hash data space, and determining whether the actual values ​​of the sharding condition are evenly distributed in the hash data space; When the actual values ​​of the sharding condition are evenly distributed, the sharding size of the corresponding sharding object is set to the default sharding size; When the actual values ​​of the sharding condition are unevenly distributed, sharding critical points are divided according to the distribution of the actual values ​​of the sharding condition, and the sharding size of the corresponding sharding object is calculated according to the sharding critical points.

2. The database sharding method according to claim 1, wherein: The extracting the operation object and operation condition corresponding to each data operation instruction in the historical data operation instruction set includes: According to the preset instruction syntax, each of the data operation instructions is marked in turn to obtain a table to be operated, a field to be operated, and a condition field; Taking the table to be operated and the field to be operated as the operation object; The condition field is used as the operation condition.

3. The database sharding method according to claim 2, wherein: The calculating the overlap between each of the operation objects includes: Assigning weights to the table to be operated and the field to be operated of each operation object according to the preset operation table weights and the preset operation field weights; Select one operation object from all operation objects in turn as a reference object, and the other operation objects as comparison objects; According to the principle that if the tables to be operated are the same, the weights of the tables to be operated are added together, and if the fields to be operated are the same, the weights of the fields to be operated are added together, the sum of the weights between the reference object and any comparison object is calculated as the degree of overlap between the reference object and the corresponding comparison object.

4. The database sharding method according to claim 2, wherein: The step of selecting the core operation condition from the operation conditions corresponding to the same shard object as the shard condition includes: Determine whether the condition field corresponding to the operation condition in each data operation instruction of the same shard object is unique; When the condition field is unique, the weight of the corresponding condition field is a preset unit value; When the condition field is not unique, the weight of each condition field is the ratio of the preset unit value to the number of condition fields; Summing the weights of the same condition field in all data operation instructions of the same shard object to obtain a total weight value of each condition field; The condition field whose total weight value is greater than a preset weight threshold is selected as the core operation condition to obtain the sharding condition.

5. The database sharding method according to claim 1, wherein: The step of sequentially publishing the shards in the shard set to corresponding servers according to the maximum load value includes: Delivering one shard in the shard set to a server in the server set in sequence, and calculating the load difference between the actual load of each server and the corresponding maximum load value; When the load difference is less than a preset difference warning value, returning to the above step of sequentially delivering a shard in the shard set to a server in the server set; When the load difference is not less than the preset difference warning value, the corresponding server is removed from the server set.

6. A database sharding device, used to execute the database sharding method according to any one of claims 1 to 5, characterized in that: The device comprises: A data operation instruction parsing module is used to obtain a historical data operation instruction set of a target database group and extract the operation object and operation condition corresponding to each data operation instruction in the historical data operation instruction set; A sharding strategy generation module is configured to calculate the overlap between each of the operation objects, treat the operation objects with an overlap greater than a preset overlap threshold as the same sharding object, select core operation conditions from the operation conditions corresponding to the same sharding object as sharding conditions, and calculate the sharding size of the sharding object based on the actual values ​​and value ranges corresponding to the sharding conditions; The sharding division and deployment module is used to shard the shard object according to the sharding conditions and the corresponding shard sizes, collect all the shards of the target database group to obtain a shard set, obtain the server set corresponding to the target database group, and the maximum load value of each server in the server set, and publish the shards in the shard set to the corresponding servers in sequence according to the maximum load value.

7. A database sharding device according to claim 6, characterized in that: The segment object recognition module calculates the overlap between each of the operation objects by the following method: Assigning weights to the table to be operated and the field to be operated of each operation object according to the preset operation table weights and the preset operation field weights; Select one operation object from all operation objects in turn as a reference object, and the other operation objects as comparison objects; According to the principle that if the tables to be operated are the same, the weights of the tables to be operated are added together, and if the fields to be operated are the same, the weights of the fields to be operated are added together, the sum of the weights between the reference object and any comparison object is calculated as the degree of overlap between the reference object and the corresponding comparison object.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the database sharding method according to any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the database sharding method according to any one of claims 1 to 5 is implemented.

Images