Data slicing method, device, equipment and medium based on artificial intelligence
Through the data sharding method based on artificial intelligence, text similarity algorithm and business type classification are used to solve the problem of performance degradation of traditional databases under large data volumes, and more efficient data call and storage are achieved.
Patent Information
- Application Number
- CN202210449787.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-04-26
AI Technical Summary
Traditional databases have a deterioration in the performance of large data volumes, and the existing data sharding methods lack flexibility and cannot effectively improve the efficiency of database data calling.
Using the data sharding method based on artificial intelligence, we obtain the configuration information of the data to be sharded, classify the data according to the business type, and calculate the semantic correlation based on the text similarity algorithm to perform sharding, and filter the hot spot fields to optimize data storage.
It improves the efficiency of database data calling, reduces the read/write pressure of a single server through flexible data sharding, and improves the level expansion function of the database.
Smart Images

Figure CN114860722B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based data slicing method, device, electronic device and storage medium. Background Art
[0002] Traditional relational databases, such as the commonly used MySQL, have performance bottlenecks, that is, when the data reaches a certain level, the performance of the database will drop significantly, and the read and write operations of the database will be affected. Data sharding is one of the common ways to optimize database performance. Since a single table / single database has a performance bottleneck for data storage, the data to be stored can be distributed to multiple database tables.
[0003] In the prior art, data is usually sharded according to a single method such as the size of the data to be sharded or the date. However, a large amount of data to be sharded corresponds to a variety of different business types, and the frequency of calling different data to be sharded in the database is not the same. Therefore, a more flexible and effective data sharding method is urgently needed to perform data sharding, thereby improving the data calling efficiency of the database. Summary of the invention
[0004] In view of the above, it is necessary to propose an artificial intelligence-based data sharding method, device, electronic device and storage medium to solve the technical problem of how to improve the data call efficiency of the database.
[0005] The present application provides a data sharding method based on artificial intelligence, the method comprising:
[0006] Obtain configuration information of the data to be sharded, the configuration information including the data volume of the data to be sharded, table creation statements, and multiple databases;
[0007] Classifying the data to be fragmented according to the business type to obtain classified data, and storing the classified data in a fragmentation table based on the configuration information as first fragmentation data;
[0008] Receive a data read request to read target data from the shard table, wherein the data read request carries a key field of the target data;
[0009] Counting the number of key fields in the historical target data to obtain the call frequency, and filtering all key fields based on the call frequency to obtain a hot field data set;
[0010] The text similarity between the first shard data and each key field in the hot field data set is calculated according to a text similarity algorithm to obtain semantic relevance, and the first shard data is segmented based on the semantic relevance to obtain second shard data.
[0011] In some embodiments, the obtaining of configuration information of the data to be sharded, the configuration information including the data volume of the data to be sharded, table creation statements, and multiple databases, includes:
[0012] The configuration information input by the user through the web interface is obtained, wherein the data volume of the data to be sharded is used to indicate how many shards the data to be sharded is split into; the table creation statement is used to generate a sharding table capable of writing the data to be sharded; and the multiple databases are used to store the data to be sharded.
[0013] In some embodiments, classifying the to-be-sharded data according to the business type to obtain classified data, and storing the classified data in the sharding table as the first sharding data based on the configuration information includes:
[0014] Different coding labels are set for the data to be fragmented for each business type according to the preset method;
[0015] The data to be sharded is classified based on the coding label to obtain classified data, and classified data of the same category is stored in the same shard table based on the configuration information, and classified data of different categories are stored in different shard tables, and all classified data stored in the shard tables are used as first shard data.
[0016] In some embodiments, the receiving of a data read request to read target data from the shard table, wherein the key field of the target data carried in the data read request includes:
[0017] Receiving a data read request to obtain a key field of the target data carried in the data read request;
[0018] Reading the routing address where the target data is located according to the pre-configured routing rules and the key field, wherein the routing address includes the shard table and database where the target data is located;
[0019] The target data is read and fed back based on the routing address.
[0020] In some embodiments, the counting of the number of key fields in the historical target data to obtain the call frequency, and filtering all key fields based on the call frequency to obtain the hotspot field data set includes:
[0021] Counting the number of each key field in the historical target data, and taking the ratio of the number of each key field to the total number of all key fields as the call frequency;
[0022] The calling frequencies are classified, and all key fields are screened according to the classification results to obtain a hotspot field data set.
[0023] In some embodiments, the classifying the call frequencies and screening all key fields according to the classification results to obtain a hotspot field data set includes:
[0024] Calculate the sum of squared errors of all call frequencies to determine the number of valid classifications for all call frequencies;
[0025] Classifying all call frequencies based on the effective classification number to obtain multiple call frequency categories;
[0026] The average value of the call frequencies contained in each call frequency category is calculated, the call frequency category corresponding to the maximum average value is selected, and the key fields corresponding to all the call frequencies in the call frequency category are used as the hot field data set.
[0027] In some embodiments, calculating the text similarity between the first shard data and each key field in the hotspot field data set according to a text similarity algorithm to obtain semantic relevance, and sharding the first shard data based on the semantic relevance to obtain second shard data includes:
[0028] Calculating the text similarity between each classification data in the first fragmented data and each key field in the hot field data set according to the text similarity algorithm;
[0029] Taking the call frequency corresponding to each key field in the hot field data set as a feature weight, and performing weighted summation on the feature weight and the text similarity to obtain the semantic relevance of each classification data in the first segmented data;
[0030] An average value of the semantic relevance of all the classified data in the first fragmented data is calculated as a fragmentation threshold, and the first fragmented data is fragmented based on the fragmentation threshold to obtain second fragmented data.
[0031] The embodiment of the present application also provides a data slicing device based on artificial intelligence, the device comprising:
[0032] An acquisition unit, used to acquire configuration information of the data to be sharded, wherein the configuration information includes the data volume of the data to be sharded, table creation statements, and multiple databases;
[0033] a classification unit, configured to classify the data to be fragmented according to the service type to obtain classified data, and store the classified data in a fragmentation table as first fragmentation data based on the configuration information;
[0034] A reading unit, configured to receive a data reading request to read target data from the shard table, wherein the data reading request carries a key field of the target data;
[0035] A screening unit, used to count the number of key fields in the historical target data to obtain a call frequency, and screen all key fields based on the call frequency to obtain a hot field data set;
[0036] A calculation unit is used to calculate the text similarity between the first shard data and each key field in the hot field data set according to a text similarity algorithm to obtain semantic relevance, and to slice the first shard data based on the semantic relevance to obtain second shard data.
[0037] The present application also provides an electronic device, the electronic device comprising:
[0038] A memory storing at least one instruction;
[0039] A processor executes instructions stored in the memory to implement the artificial intelligence-based data slicing method.
[0040] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the artificial intelligence-based data sharding method.
[0041] The present application screens all key fields by counting the number of key fields in the historical target data, then calculates the text similarity between the data to be sharded and the hot field data set to obtain the semantic relevance, and shards the classified data based on the semantic relevance. The hot data can be flexibly screened and the hot data in the data to be sharded can be stored in separate tables, thereby improving the data call efficiency of the database. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a flowchart of a preferred embodiment of the artificial intelligence-based data sharding method involved in this application.
[0043] Figure 2 It is a functional module diagram of a preferred embodiment of the artificial intelligence-based data slicing device involved in this application.
[0044] Figure 3 It is a structural schematic diagram of an electronic device of a preferred embodiment of the artificial intelligence-based data slicing method involved in the present application.
[0045] Figure 4 This is a graph showing the relationship between SSE and k involved in this application. DETAILED DESCRIPTION
[0046] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.
[0047] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0049] The embodiment of the present application provides an artificial intelligence-based data slicing method, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0050] An electronic device can be any electronic product that can interact with a customer, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.
[0051] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.
[0052] The network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.
[0053] like Figure 1 As shown, it is a flow chart of a preferred embodiment of the data slicing method based on artificial intelligence of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.
[0054] S10, obtaining configuration information of the data to be sharded, where the configuration information includes the data volume of the data to be sharded, table creation statements, and multiple databases.
[0055] In an optional embodiment, the obtaining of configuration information of the data to be sharded, the configuration information including the data volume of the data to be sharded, table creation statements, and multiple databases, includes:
[0056] The configuration information input by the user through the web interface is obtained, wherein the data volume of the data to be sharded is used to indicate how many shards the data to be sharded is split into; the table creation statement is used to generate a sharding table capable of writing the data to be sharded; and the multiple databases are used to store the data to be sharded.
[0057] In this optional embodiment, a web interface may be provided in advance to obtain configuration information input by the user through the web interface. The data volume of the data to be sharded is a unit size, and the data to be sharded needs to be split into multiple shards of the same size. For example, when a data to be sharded is 1000GB, and the data volume of the sharded data pre-configured in the web interface is 50GB, the data to be sharded can be split into 20 shards of 50GB each.
[0058] In this optional embodiment, a table creation statement input by a user in a web interface is obtained. The table creation statement is used to generate a sharding table with a uniform format, and an index column for sharding in the sharding table can be correspondingly specified. The index column is used to store all index data in the sharding table in the form of a column, and the index data can be a key field of the data to be sharded. In addition, the sharding table is used to write the sharding data, so as to obtain a sharding table that can maintain a uniform format when storing data in different databases.
[0059] In this optional embodiment, multiple databases input by the user in the web interface are obtained. Among them, the database is a "warehouse" designated by the user for storing the data to be sharded, and its storage object is in the form of a table. Therefore, in this embodiment of the application, the data to be sharded needs to be written into the sharding table, and then the sharding table is routed and stored in different databases.
[0060] In this way, the configuration information of the data to be sharded can be quickly obtained based on the web interface, providing data support for subsequent processes while improving the efficiency of human-computer interaction.
[0061] S11, classifying the data to be fragmented according to the business type to obtain classified data, and storing the classified data in a fragmentation table based on the configuration information as first fragmentation data.
[0062] In an optional embodiment, classifying the to-be-sharded data according to the business type to obtain classified data, and storing the classified data in the sharding table as the first sharding data based on the configuration information includes:
[0063] S111, setting different coding labels for the to-be-segmented data of each service type according to a preset method.
[0064] S112, classify the data to be sharded based on the coding label to obtain classified data, and store the classified data of the same category in the same shard table based on the configuration information, and store the classified data of different categories in different shard tables, and use all the classified data stored in the shard tables as the first shard data.
[0065] In this optional embodiment, when the application continuously executes business functions, it will continuously write the data to be sharded; these data to be sharded are specifically related to the business launched by the application, and may be needed in subsequent business, and have important utilization value, so they need to be stored. The commonly used storage method is database storage, such as MYSQL, MYCAT and other commonly used data.
[0066] In this optional embodiment, different coding labels may be set for the to-be-segmented data of each service type according to a preset method. The coding labels may be letters, numbers or special symbols, and this solution does not impose any specific restrictions on this.
[0067] In this optional embodiment, the data to be sharded with the same coding label are classified into the same category, thereby obtaining classified data of multiple categories, and then after these classified classified data are split into multiple shard data according to the data volume, the multiple shard data are stored in multiple databases. In the setting, the object used by the database to store data is a table, so the shard data needs to be written into the table before the corresponding storage business can be performed. Therefore, the corresponding shard table can be generated according to the table creation statement, and then the shard data belonging to the same category are respectively written into the corresponding shard table generated in advance, and multiple target shard tables are obtained, so that the shard data is stored in the database in the form of a shard table, and all the classified data stored in the shard table is used as the first shard data.
[0068] In this optional embodiment, multiple target sharding tables are sequentially stored in one of the multiple databases. If there are still target sharding tables that have not been stored in the database, the remaining target sharding tables in the target sharding tables are continuously stored in a database other than the database storing the target sharding tables in the multiple databases until all the target sharding tables are stored in the database.
[0069] In this optional embodiment, since the amount of data in a shard data is usually very large, a shard table may not be able to write a complete shard data. Therefore, multiple shard tables can be used to write the same shard data, and the tables are distinguished by serial numbers. For example: shard table 1 to shard table 5000 are used to write the first shard data; shard table 5001 to shard table 10000 are used to write the second shard data... and so on. Among them, the same shard table only stores shard data with the same encoding label.
[0070] If the shard tables to which the first shard data is written include shard tables 1 to shard table 5000, theoretically, according to the principle of "one database, one table", these 5000 shard tables should all be stored in the same database. However, due to the limited database capacity, if the amount of data in the shard tables to be stored exceeds the database capacity, the shard tables that exceed the capacity need to be stored in another database. Of course, this method is only an optional backup operation, because the database capacity is currently huge and also supports expansion operations.
[0071] In this way, the amount of data to be sharded can be effectively dispersed, reducing the read / write pressure faced by a single server resource, and enabling the database to have good horizontal expansion capabilities.
[0072] S12, receiving a data read request to read target data from the shard table, wherein the data read request carries a key field of the target data.
[0073] In an optional embodiment, the receiving a data read request to read the target data from the shard table, wherein the key field of the target data carried in the data read request includes:
[0074] S121, receiving a data read request to obtain a key field of target data carried in the data read request.
[0075] S122, reading the routing address where the target data is located according to the pre-configured routing rules and the key field, wherein the routing address includes the shard table and database where the target data is located.
[0076] S123, reading and feeding back the target data based on the routing address.
[0077] In this optional embodiment, when the application currently needs to read the target data due to business needs, a corresponding data read request can be issued, wherein the target data refers to a certain item of data in all classified data. The data read request carries the key field of the target data, which can be obtained by parsing the data read request. At the same time, the key field can be used as the index of the target data, that is, the corresponding target data can be directly found through the index.
[0078] In this optional embodiment, based on the key field, the routing address where the target data is located is read through the pre-configured routing rules, and the routing address includes the shard table and database where the target data is located. Among them, the route is equivalent to a road, and the routing rule is equivalent to a signpost, which is used to instruct the application how to obtain the target data in the corresponding database and shard table, so it is necessary to set the corresponding routing rules in advance by writing script commands.
[0079] This solution provides a routing rule engine for database cache management. ThingsBoard rule engine can be used in this solution. Among them, the main logical unit of the rule engine is the rule node. There is a correlation between rule nodes. The rule node can route the message to the next specified node to complete the data reading work. Therefore, when the data is read later, the routing address where the target data is located, that is, the location of the shard table and the database where the target data is located, can be read based on the key fields obtained by parsing the request, using the routing rules, and the target data can be read and fed back according to the routing address.
[0080] In this optional embodiment, after the target data is read, the target data is fed back to the corresponding application. It should be noted that, in special cases, when the application needs to read multiple target data, and they are stored in different shard tables, it is necessary to perform aggregation operations on the multiple target data read to obtain the aggregated target data. Therefore, the embodiment of the present application provides a sql parsing aggregation engine, which is responsible for parsing and processing the sql request of the processing application, and splitting the processed sql instructions according to the routing rule engine, concurrently processing multiple shard tables and multiple databases, performing aggregation operations after obtaining the execution results, and finally returning the data or operation results required by the application.
[0081] In this way, the target data can be quickly acquired according to the key fields carried in the data reading request, thereby improving the efficiency of acquiring the target data. At the same time, data support can be provided for subsequent process analysis of hot spot data according to the key fields.
[0082] S13, counting the number of key fields in the historical target data to obtain a call frequency, and screening all key fields based on the call frequency to obtain a hotspot field data set.
[0083] In an optional embodiment, the counting of the number of key fields in the historical target data to obtain the call frequency, and screening all key fields based on the call frequency to obtain the hot field data set includes:
[0084] S131, counting the number of each key field in the historical target data, and taking the ratio of the number of each key field to the total number of all key fields as the calling frequency.
[0085] S132, classifying the calling frequencies, and screening all key fields according to the classification results to obtain a hotspot field data set.
[0086] In this optional embodiment, the number of each key field in the historical target data obtained by the application is first counted, and the ratio of the number of each key field to the total number of all key fields is used as the call frequency of the corresponding key field.
[0087] In this optional embodiment, the call frequency can be classified by the K-means clustering algorithm. Among them, the K-means clustering algorithm is an iterative clustering analysis algorithm, and its steps are to pre-divide all the call frequency data into k groups, and randomly select k call frequency data as the initial cluster centers, and then calculate the distance between each call frequency data and each cluster center, and assign each call frequency data to the cluster center closest to it. The cluster center and all the call frequency data assigned to the cluster center represent a cluster. Every time a call frequency data is assigned, the cluster center of the cluster where the call frequency data is located will be recalculated based on all the existing call frequency data in the cluster. This process will be repeated until a certain termination condition is met. The termination condition can be that no (or a minimum number) of call frequency data are reallocated to different clusters, no (or a minimum number) of cluster centers change again, and the error square sum is locally minimized.
[0088] In this optional embodiment, the premise of using the K-means clustering algorithm is to specify the number of categories k of the call frequency data. As the number of categories increases, the call frequency will be divided more finely, the aggregation degree of each category will gradually increase, and the corresponding error square sum SSE will gradually decrease. Therefore, in this solution, the corresponding error square sum SSE under different numbers of categories is calculated to obtain a reasonable number of categories k. The calculation formula for the error square sum is:
[0089]
[0090] Among them, SSE is the sum of squared errors, C i is the i-th call frequency category, k is the preset number of call frequency categories, and p is the call frequency category C i The data points in Indicates call frequency category C i The mean of all call frequency data points in .
[0091] For example, let k be 1, and the corresponding call frequency data be (10, 20, 30). Then the average value of all data points is 20, and SSE = (10-20). 2 +(20-20) 2 +(30-20) 2 =200, then the calculated sum of squared errors is 200.
[0092] In this optional embodiment, when k is less than the ideal number of categories, the increase in k will greatly increase the degree of aggregation of each call frequency category, so the decrease in SSE will be large. When k reaches the ideal number of categories, the degree of aggregation obtained by further increasing k will rapidly decrease, so the decrease in SSE will drop sharply, and then tend to be flat as the k value continues to increase. Therefore, the k value corresponding to the data point with the highest curvature can be determined by establishing a relationship graph of SSE increasing with k. The k value at this time is the ideal number of call frequency categories.
[0093] For example, Figure 4 In the graph showing the relationship between SSE and k, when k=4, the curvature of the corresponding data point is the largest, so the optimal number of categories is k=4.
[0094] In this optional embodiment, the ideal number of call frequency categories k finally determined is used as the effective number of categories, and the call frequency is classified into multiple call frequency categories by using a K-means clustering algorithm based on the effective number of categories.
[0095] In this optional embodiment, after calculating the average value of the call frequencies contained in each call frequency category and selecting the call frequency category corresponding to the maximum average value, the key fields corresponding to all the call frequencies in the call frequency category are used as the hot field data set.
[0096] In this way, the hot data in the classified data can be screened out, which is convenient for subsequent processes to perform table storage accordingly, thereby accurately sharding the data to be sharded.
[0097] S14, calculating the text similarity between the first shard data and each key field in the hot field data set according to a text similarity algorithm to obtain semantic relevance, and sharding the first shard data based on the semantic relevance to obtain second shard data.
[0098] In an optional embodiment, calculating the text similarity between the first shard data and each key field in the hotspot field data set according to a text similarity algorithm to obtain semantic relevance, and sharding the first shard data based on the semantic relevance to obtain second shard data includes:
[0099] S141, calculating the text similarity between each classification data in the first fragmented data and each key field in the hot field data set according to a text similarity algorithm.
[0100] S142, taking the call frequency corresponding to each key field in the hot field data set as a feature weight, and performing weighted summation on the feature weight and the text similarity to obtain the semantic relevance of each classification data in the first segmented data.
[0101] S143, calculating an average value of the semantic relevance of all the classified data in the first fragmented data as a fragmentation threshold, and fragmenting the first fragmented data based on the fragmentation threshold to obtain second fragmented data.
[0102] In this optional embodiment, the text similarity algorithm may use a vector space model (VSM) algorithm. VSM formalizes each data in the first segmented data into a point in space and gives it in the form of a vector, thereby simplifying the processing of the first segmented data into vector operations in the vector space, thereby reducing the complexity of calculating the text similarity of the classified data in the first segmented data.
[0103] In this optional embodiment, the text similarity between each classified data in the first shard data and each key field in the hot field data set is calculated respectively by the VSM algorithm, and then the call frequency corresponding to each key field in the hot field data set is used as the feature weight, and the feature weight and the text similarity are weighted and summed to obtain the semantic relevance of each classified data in the first shard data.
[0104] For example, there are 4 key fields A, B, C, and D in the hot data set. The text similarities between the classified data E in the first shard data and the 4 key fields A, B, C, and D are calculated by the VSM algorithm to be 0.6, 0.2, 0.1, and 0.3, respectively. The call frequencies corresponding to the 4 key fields A, B, C, and D are 0.8, 0.5, 0.6, and 0.2, and the semantic relevance of the classified data E in the first shard data is calculated by weighted summation to be 0.6×0.8+0.2×0.5+0.1×0.6+0.3×0.2=0.7.
[0105] In this optional embodiment, the average value of the semantic relevance of all the classified data in the first shard data is calculated as the shard threshold, and the first shard data is sharded based on the shard threshold to obtain the second shard data. The specific process is: split each shard table into a data master table and a data slave table, and write the classified data greater than the shard threshold into the data slave table, and write the classified data not greater than the shard threshold into the data master table. When splitting each shard table, each shard table can be used as the data master table, and a shard table that does not contain any data can be selected as the data slave table of the shard table. Finally, all the first shard data written into the data master table and the data slave table are used as the second shard data, thereby completing the data sharding process for the sharded data.
[0106] In this way, hot data can be flexibly screened and hot data in classified data can be stored in separate tables, thereby improving the data call efficiency of the database.
[0107] See also Figure 2 , Figure 2 1 is a functional module diagram of a preferred embodiment of the data slicing device based on artificial intelligence of the present application. The data slicing device based on artificial intelligence 11 includes an acquisition unit 110, a classification unit 111, a reading unit 112, a screening unit 113, and a calculation unit 114. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and can complete fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.
[0108] In an optional embodiment, the acquisition unit 110 is used to acquire configuration information of the data to be sharded, where the configuration information includes the data volume of the data to be sharded, table creation statements, and multiple databases.
[0109] In an optional embodiment, the obtaining of configuration information of the data to be sharded, the configuration information including the data volume of the data to be sharded, table creation statements, and multiple databases, includes:
[0110] The configuration information input by the user through the web interface is obtained, wherein the data volume of the data to be sharded is used to indicate how many shards the data to be sharded is split into; the table creation statement is used to generate a sharding table capable of writing the data to be sharded; and the multiple databases are used to store the data to be sharded.
[0111] In this optional embodiment, a web interface may be provided in advance to obtain configuration information input by the user through the web interface. The data volume of the data to be sharded is a unit size, and the data to be sharded needs to be split into multiple shards of the same size. For example, when a data to be sharded is 1000GB, and the data volume of the sharded data pre-configured in the web interface is 50GB, the data to be sharded can be split into 20 shards of 50GB each.
[0112] In this optional embodiment, a table creation statement input by the user in the web interface is obtained. The table creation statement is used to generate a sharding table with a unified format, and the index column used for sharding in the sharding table can be correspondingly specified. The index column contains multiple indexes, and the index is a special database structure that is a separate, physical sort of the values of one or more columns in the database table. It is a collection of one or more column values in a table and a corresponding list of logical pointers pointing to the data pages that physically identify these values in the table. The role of the index is equivalent to the catalog of a book, and the required content can be quickly found according to the page number in the catalog. In addition, the sharding table is used to write sharding data to obtain a sharding table that can maintain a unified format when storing data in different databases.
[0113] In this optional embodiment, multiple databases input by the user in the web interface are obtained. Among them, the database is a "warehouse" designated by the user for storing the data to be sharded, and its storage object is in the form of a table. Therefore, in this embodiment of the application, the data to be sharded needs to be written into the sharding table, and then the sharding table is routed and stored in different databases.
[0114] In an optional embodiment, the classification unit 111 is used to classify the data to be fragmented according to the business type to obtain classified data, and store the classified data in the fragmentation table as the first fragmentation data based on the configuration information.
[0115] In an optional embodiment, classifying the to-be-sharded data according to the business type to obtain classified data, and storing the classified data in the sharding table as the first sharding data based on the configuration information includes:
[0116] Different coding labels are set for the data to be fragmented for each business type according to the preset method;
[0117] The data to be sharded is classified based on the coding label to obtain classified data, and classified data of the same category is stored in the same shard table based on the configuration information, and classified data of different categories are stored in different shard tables, and all classified data stored in the shard tables are used as first shard data.
[0118] In this optional embodiment, when the application continuously executes business functions, it will continuously write the data to be sharded; these data to be sharded are specifically related to the business launched by the application, and may be needed in subsequent business, and have important utilization value, so they need to be stored. The commonly used storage method is database storage, such as MYSQL, MYCAT and other commonly used data.
[0119] In this optional embodiment, different coding labels may be set for the to-be-segmented data of each service type according to a preset method. The coding labels may be letters, numbers or special symbols, and this solution does not impose any specific restrictions on this.
[0120] In this optional embodiment, the data to be sharded with the same coding label are classified into the same category, thereby obtaining classified data of multiple categories, and then after these classified classified data are split into multiple shard data according to the data volume, the multiple shard data are stored in multiple databases. In the setting, the object used by the database to store data is a table, so the shard data needs to be written into the table before the corresponding storage business can be performed. Therefore, the corresponding shard table can be generated according to the table creation statement, and then the shard data belonging to the same category are respectively written into the corresponding shard table generated in advance, and multiple target shard tables are obtained, so that the shard data is stored in the database in the form of a shard table, and all the classified data stored in the shard table is used as the first shard data.
[0121] In this optional embodiment, multiple target sharding tables are sequentially stored in one of the multiple databases. If there are still target sharding tables that have not been stored in the database, the remaining target sharding tables in the target sharding tables are continuously stored in a database other than the database storing the target sharding tables in the multiple databases until all the target sharding tables are stored in the database.
[0122] In this optional embodiment, since the amount of data in a shard data is usually very large, a shard table may not be able to write a complete shard data. Therefore, multiple shard tables can be used to write the same shard data, and the tables are distinguished by serial numbers. For example: shard table 1 to shard table 5000 are used to write the first shard data; shard table 5001 to shard table 10000 are used to write the second shard data... and so on. Among them, the same shard table only stores shard data with the same encoding label.
[0123] If the shard tables to which the first shard data is written include shard tables 1 to shard table 5000, theoretically, according to the principle of "one database, one table", these 5000 shard tables should all be stored in the same database. However, due to the limited database capacity, if the amount of data in the shard tables to be stored exceeds the database capacity, the shard tables that exceed the capacity need to be stored in another database. Of course, this method is only an optional backup operation, because the database capacity is currently huge and also supports expansion operations.
[0124] In an optional embodiment, the reading unit 112 is used to receive a data reading request to read the target data from the shard table, and the data reading request carries a key field of the target data.
[0125] In an optional embodiment, the receiving a data read request to read the target data from the shard table, wherein the key field of the target data carried in the data read request includes:
[0126] Receiving a data read request to obtain a key field of the target data carried in the data read request;
[0127] Reading the routing address where the target data is located according to the pre-configured routing rules and the key field, wherein the routing address includes the shard table and database where the target data is located;
[0128] The target data is read and fed back based on the routing address.
[0129] In this optional embodiment, when the application currently needs to read the target data due to business needs, a corresponding data read request can be issued, wherein the target data refers to a certain item of data in all classified data. The data read request carries the key field of the target data, which can be obtained by parsing the data read request. At the same time, the key field can be used as the index of the target data, that is, the corresponding target data can be directly found through the index.
[0130] In this optional embodiment, based on the key field, the routing address where the target data is located is read through the pre-configured routing rules, and the routing address includes the shard table and database where the target data is located. Among them, the route is equivalent to a road, and the routing rule is equivalent to a signpost, which is used to instruct the application how to obtain the target data in the corresponding database and shard table, so it is necessary to set the corresponding routing rules in advance by writing script commands.
[0131] This solution provides a routing rule engine for database cache management. ThingsBoard rule engine can be used in this solution. Among them, the main logical unit of the rule engine is the rule node. There is a correlation between rule nodes. The rule node can route the message to the next specified node to complete the data reading work. Therefore, when the data is read later, the routing address where the target data is located, that is, the location of the shard table and the database where the target data is located, can be read based on the key fields obtained by parsing the request, using the routing rules, and the target data can be read and fed back according to the routing address.
[0132] In this optional embodiment, after the target data is read, the target data is fed back to the corresponding application. It should be noted that, in special cases, when the application needs to read multiple target data, and they are stored in different shard tables, it is necessary to perform aggregation operations on the multiple target data read to obtain the aggregated target data. Therefore, the embodiment of the present application provides a sql parsing aggregation engine, which is responsible for parsing and processing the sql request of the processing application, and splitting the processed sql instructions according to the routing rule engine, concurrently processing multiple shard tables and multiple databases, performing aggregation operations after obtaining the execution results, and finally returning the data or operation results required by the application.
[0133] In an optional embodiment, the screening unit 113 is used to count the number of key fields in the historical target data to obtain the calling frequency, and screen all the key fields based on the calling frequency to obtain the hot field data set.
[0134] In an optional embodiment, the counting of the number of key fields in the historical target data to obtain the call frequency, and screening all key fields based on the call frequency to obtain the hot field data set includes:
[0135] Counting the number of each key field in the historical target data, and taking the ratio of the number of each key field to the total number of all key fields as the call frequency;
[0136] The calling frequencies are classified, and all key fields are screened according to the classification results to obtain a hotspot field data set.
[0137] In this optional embodiment, the number of each key field in the historical target data obtained by the application is first counted, and the ratio of the number of each key field to the total number of all key fields is used as the call frequency of the corresponding key field.
[0138] In this optional embodiment, the call frequency can be classified by the K-means clustering algorithm. Among them, the K-means clustering algorithm is an iterative clustering analysis algorithm, and its steps are to pre-divide all the call frequency data into k groups, and randomly select k call frequency data as the initial cluster centers, and then calculate the distance between each call frequency data and each cluster center, and assign each call frequency data to the cluster center closest to it. The cluster center and all the call frequency data assigned to the cluster center represent a cluster. Every time a call frequency data is assigned, the cluster center of the cluster where the call frequency data is located will be recalculated based on all the existing call frequency data in the cluster. This process will be repeated until a certain termination condition is met. The termination condition can be that no (or a minimum number) of call frequency data are reallocated to different clusters, no (or a minimum number) of cluster centers change again, and the error square sum is locally minimized.
[0139] In this optional embodiment, the premise of using the K-means clustering algorithm is to specify the number of categories k of the call frequency data. As the number of categories increases, the call frequency will be divided more finely, the aggregation degree of each category will gradually increase, and the corresponding error square sum SSE will gradually decrease. Therefore, in this solution, the corresponding error square sum SSE under different numbers of categories is calculated to obtain a reasonable number of categories k. The calculation formula for the error square sum is:
[0140]
[0141] Among them, SSE is the sum of squared errors, C i is the i-th call frequency category, k is the preset number of call frequency categories, and p is the call frequency category C i The data points in Indicates call frequency category C i The mean of all call frequency data points in .
[0142] For example, let k be 1, and the corresponding call frequency data be (10, 20, 30). Then the average value of all data points is 20, and SSE = (10-20). 2 +(20-20) 2 +(30-20) 2 =200, then the calculated sum of squared errors is 200.
[0143] In this optional embodiment, when k is less than the ideal number of categories, the increase in k will greatly increase the degree of aggregation of each call frequency category, so the decrease in SSE will be large. When k reaches the ideal number of categories, the degree of aggregation obtained by further increasing k will rapidly decrease, so the decrease in SSE will drop sharply, and then tend to be flat as the k value continues to increase. Therefore, the k value corresponding to the data point with the highest curvature can be determined by establishing a relationship graph of SSE increasing with k. The k value at this time is the ideal number of call frequency categories.
[0144] For example, Figure 4 In the graph showing the relationship between SSE and k, when k=4, the curvature of the corresponding data point is the largest, so the optimal number of categories is k=4.
[0145] In this optional embodiment, the ideal number of call frequency categories k finally determined is used as the effective number of categories, and the call frequency is classified into multiple call frequency categories by using a K-means clustering algorithm based on the effective number of categories.
[0146] In this optional embodiment, after calculating the average value of the call frequencies contained in each call frequency category and selecting the call frequency category corresponding to the maximum average value, the key fields corresponding to all the call frequencies in the call frequency category are used as the hot field data set.
[0147] In an optional embodiment, the calculation unit 114 is used to calculate the text similarity between the first shard data and each key field in the hot field data set according to a text similarity algorithm to obtain semantic relevance, and to segment the first shard data based on the semantic relevance to obtain second shard data.
[0148] In an optional embodiment, calculating the text similarity between the first shard data and each key field in the hotspot field data set according to a text similarity algorithm to obtain semantic relevance, and sharding the first shard data based on the semantic relevance to obtain second shard data includes:
[0149] Calculating the text similarity between each classification data in the first fragmented data and each key field in the hot field data set according to the text similarity algorithm;
[0150] Taking the call frequency corresponding to each key field in the hot field data set as a feature weight, and performing weighted summation on the feature weight and the text similarity to obtain the semantic relevance of each classification data in the first segmented data;
[0151] An average value of the semantic relevance of all the classified data in the first fragmented data is calculated as a fragmentation threshold, and the first fragmented data is fragmented based on the fragmentation threshold to obtain second fragmented data.
[0152] In this optional embodiment, the text similarity algorithm may use a vector space model (VSM) algorithm. VSM formalizes each data in the first segmented data into a point in space and gives it in the form of a vector, thereby simplifying the processing of the first segmented data into vector operations in the vector space, thereby reducing the complexity of calculating the text similarity of the classified data in the first segmented data.
[0153] In this optional embodiment, the text similarity between each classified data in the first shard data and each key field in the hot field data set is calculated respectively by the VSM algorithm, and then the call frequency corresponding to each key field in the hot field data set is used as the feature weight, and the feature weight and the text similarity are weighted and summed to obtain the semantic relevance of each classified data in the first shard data.
[0154] For example, there are 4 key fields A, B, C, and D in the hot data set. The text similarities between the classified data E in the first shard data and the 4 key fields A, B, C, and D are calculated by the VSM algorithm to be 0.6, 0.2, 0.1, and 0.3, respectively. The call frequencies corresponding to the 4 key fields A, B, C, and D are 0.8, 0.5, 0.6, and 0.2, and the semantic relevance of the classified data E in the first shard data is calculated by weighted summation to be 0.6×0.8+0.2×0.5+0.1×0.6+0.3×0.2=0.7.
[0155] In this optional embodiment, the average value of the semantic relevance of all the classified data in the first shard data is calculated as the shard threshold, and the first shard data is sharded based on the shard threshold to obtain the second shard data. The specific process is: split each shard table into a data master table and a data slave table, and write the classified data greater than the shard threshold into the data slave table, and write the classified data not greater than the shard threshold into the data master table. When splitting each shard table, each shard table can be used as the data master table, and a shard table that does not contain any data can be selected as the data slave table of the shard table. Finally, all the first shard data written into the data master table and the data slave table are used as the second shard data, thereby completing the data sharding process for the sharded data.
[0156] It can be seen from the above technical solutions that the present application can screen all key fields by counting the number of key fields in the historical target data, and then calculate the text similarity between the data to be sharded and the hot field data set to obtain the semantic relevance, and shard the classified data based on the semantic relevance. It can flexibly screen the hot data and store the hot data in the data to be sharded in separate tables, thereby improving the data call efficiency of the database.
[0157] See also Figure 3 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement the data slicing method based on artificial intelligence described in any of the above embodiments.
[0158] In an optional embodiment, the electronic device 1 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13, such as an artificial intelligence-based data slicing program.
[0159] Figure 3 Only the electronic device 1 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0160] Combination Figure 1 The memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement a data slicing method based on artificial intelligence, and the processor 13 can execute the plurality of instructions to implement:
[0161] Obtain configuration information of the data to be sharded, the configuration information including the data volume of the data to be sharded, table creation statements, and multiple databases;
[0162] Classifying the data to be fragmented according to the business type to obtain classified data, and storing the classified data in a fragmentation table based on the configuration information as first fragmentation data;
[0163] Receive a data read request to read target data from the shard table, wherein the data read request carries a key field of the target data;
[0164] Counting the number of key fields in the historical target data to obtain the call frequency, and filtering all key fields based on the call frequency to obtain a hot field data set;
[0165] The text similarity between the first shard data and each key field in the hot field data set is calculated according to a text similarity algorithm to obtain semantic relevance, and the first shard data is segmented based on the semantic relevance to obtain second shard data.
[0166] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.
[0167] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may be a bus structure or a star structure. The electronic device 1 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 1 may also include input and output devices, network access devices, etc.
[0168] It should be noted that the electronic device 1 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.
[0169] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the data segmentation program based on artificial intelligence, but also can be used to temporarily store data that has been output or is to be output.
[0170] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, and executes or executes programs or modules stored in the memory 12 (for example, executing data slicing programs based on artificial intelligence, etc.), and calls the data stored in the memory 12 to execute various functions of the electronic device 1 and process data.
[0171] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the data slicing method based on artificial intelligence, for example Figure 1 Steps shown.
[0172] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an acquisition unit 110, a classification unit 111, a reading unit 112, a screening unit 113, and a calculation unit 114.
[0173] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of the data slicing method based on artificial intelligence described in each embodiment of the present application.
[0174] If the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.
[0175] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.
[0176] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.
[0177] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0178] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Figure 3 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.
[0179] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the artificial intelligence-based data segmentation method described in any of the above embodiments.
[0180] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0181] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0182] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0183] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.
[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.
Claims
1. A data sharding method based on artificial intelligence, It is characterized in that The method comprises: Obtain configuration information of the data to be sharded, the configuration information including the data volume of the data to be sharded, table creation statements, and multiple databases; Classifying the data to be fragmented according to the business type to obtain classified data, and storing the classified data in a fragmentation table based on the configuration information as first fragmentation data; Receive a data read request to read target data from the shard table, wherein the data read request carries a key field of the target data; Counting the number of key fields in historical target data to obtain the call frequency, and filtering all key fields based on the call frequency to obtain a hot field data set, including: counting the number of key fields in historical target data, and taking the ratio of the number of each key field to the total number of all key fields as the call frequency; classifying the call frequency, and filtering all key fields according to the classification result to obtain a hot field data set; wherein, classifying the call frequency, and filtering all key fields according to the classification result to obtain a hot field data set includes: calculating the sum of square errors of all call frequencies to determine the effective classification number of all call frequencies; classifying all call frequencies based on the effective classification number to obtain multiple call frequency categories; calculating the average value of the call frequencies contained in each call frequency category, selecting the call frequency category corresponding to the maximum average value, and taking the key fields corresponding to all call frequencies in the call frequency category as the hot field data set; The method comprises: calculating the text similarity between the first shard data and each key field in the hot field data set according to a text similarity algorithm to obtain semantic relevance, and slicing the first shard data based on the semantic relevance to obtain second shard data, including: calculating the text similarity between each classified data in the first shard data and each key field in the hot field data set according to a text similarity algorithm; taking the call frequency corresponding to each key field in the hot field data set as a feature weight, and performing weighted summation of the feature weight and the text similarity to obtain the semantic relevance of each classified data in the first shard data; calculating the average value of the semantic relevance of all the classified data in the first shard data as a sharding threshold, and slicing the first shard data based on the sharding threshold to obtain second shard data.
2. The artificial intelligence-based data sharding method according to claim 1, It is characterized in that The step of obtaining configuration information of the data to be sharded, wherein the configuration information includes the data volume of the data to be sharded, table creation statements, and multiple databases, includes: The configuration information input by the user through the web interface is obtained, wherein the data volume of the data to be sharded is used to indicate how many shards the data to be sharded is split into; the table creation statement is used to generate a sharding table capable of writing the data to be sharded; and the multiple databases are used to store the data to be sharded.
3. The data sharding method based on artificial intelligence as claimed in claim 1, It is characterized in that The classifying the to-be-sharded data according to the business type to obtain the classified data, and storing the classified data in the sharding table as the first sharding data based on the configuration information includes: Different coding labels are set for the data to be fragmented for each business type according to the preset method; The data to be sharded is classified based on the coding label to obtain classified data, and classified data of the same category is stored in the same shard table based on the configuration information, and classified data of different categories are stored in different shard tables, and all classified data stored in the shard tables are used as first shard data.
4. The artificial intelligence-based data sharding method according to claim 1, It is characterized in that The receiving of a data read request to read target data from the shard table, wherein the key fields of the target data carried in the data read request include: Receiving a data read request to obtain a key field of the target data carried in the data read request; Reading the routing address where the target data is located according to the pre-configured routing rules and the key field, wherein the routing address includes the shard table and database where the target data is located; The target data is read and fed back based on the routing address.
5. A data slicing device based on artificial intelligence, It is characterized in that The device comprises: An acquisition unit, used to acquire configuration information of the data to be sharded, wherein the configuration information includes the data volume of the data to be sharded, table creation statements, and multiple databases; a classification unit, configured to classify the data to be fragmented according to the service type to obtain classified data, and store the classified data in a fragmentation table as first fragmentation data based on the configuration information; A reading unit, configured to receive a data reading request to read target data from the shard table, wherein the data reading request carries a key field of the target data; A screening unit, used for counting the number of each key field in the historical target data to obtain the call frequency, and screening all the key fields based on the call frequency to obtain a hot field data set, including: counting the number of each key field in the historical target data, and taking the ratio of the number of each key field to the total number of all key fields as the call frequency; classifying the call frequency, and screening all the key fields according to the classification result to obtain the hot field data set; wherein, classifying the call frequency, and screening all the key fields according to the classification result to obtain the hot field data set includes: calculating the sum of square errors of all the call frequencies to determine the effective classification number of all the call frequencies; classifying all the call frequencies based on the effective classification number to obtain multiple call frequency categories; calculating the average value of the call frequencies contained in each call frequency category, selecting the call frequency category corresponding to the maximum average value, and taking the key fields corresponding to all the call frequencies in the call frequency category as the hot field data set; A calculation unit is used to calculate the text similarity between the first shard data and each key field in the hot field data set according to a text similarity algorithm to obtain semantic relevance, and to slice the first shard data based on the semantic relevance to obtain second shard data, including: calculating the text similarity between each classified data in the first shard data and each key field in the hot field data set according to the text similarity algorithm; taking the call frequency corresponding to each key field in the hot field data set as a feature weight, and weighted summing the feature weight and the text similarity to obtain the semantic relevance of each classified data in the first shard data; calculating the average value of the semantic relevance of all classified data in the first shard data as a shard threshold, and sharding the first shard data based on the shard threshold to obtain second shard data.
6. An electronic device, It is characterized in that The electronic device comprises: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the artificial intelligence-based data sharding method as described in any one of claims 1 to 4.
7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the artificial intelligence-based data slicing method as described in any one of claims 1 to 4.
