Model training method and device, fragmentation method and device, electronic equipment, chip, storage medium and computer program product

Abstract

The embodiment of the invention provides a model training method, a fragmentation method, a model training device, a fragmentation device, electronic equipment, a chip, a storage medium and a computer program product. The model training method comprises the following steps: constructing a training set; the training set comprises data of a plurality of fragments; the dimensions of the data in the training set comprise one or more of the following dimensions: query frequency, write-in frequency, cognitive load, traceability analysis, co-occurrence times, user preference, knowledge measurement indexes and abnormal requests; training a first model through the training set to obtain a trained first model; the first model is a fuzzy reasoning multi-classification model, and the first model is used for determining fragments to which data input into the first model belongs.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of database technology, and in particular, to a model training method, a sharding method, a model training device, a sharding device, an electronic device, a chip, a storage medium, and a computer program product. Background Art

[0002] As the business scale of the database system expands, the requirements for the performance, capacity, and data transmission efficiency of the database continue to increase. Sharding technology has become an important solution. However, due to the continuous change of the load of each shard, the shards need to be continuously adjusted. In related technologies, a new database shard instance is created by determining whether the data threshold stored in the loads of multiple database nodes is skewed, so as to provide storage space for the database node loads that need to be redistributed. However, this adjustment method will perform frequent sharding, resulting in a large number of small fragments in the database and increasing the query cost. Summary of the Invention

[0003] Embodiments of the present application provide a model training method, a sharding method, a model training device, a sharding device, an electronic device, a chip, a storage medium, and a computer program product.

[0004] The model training method provided by the embodiments of the present application includes:

[0005] Construct a training set; the training set includes data of multiple shards; the dimensions of the data in the training set include one or more of the following dimensions: query frequency, write frequency, cognitive load, traceability analysis, co-occurrence times, user preference, knowledge measurement index, and abnormal request;

[0006] Train a first model through the training set to obtain a trained first model; the first model is a fuzzy inference multi-classification model, and the first model is used to determine the shard to which the data input into the first model belongs.

[0007] The sharding method provided by the embodiments of the present application includes:

[0008] Monitor each shard to identify the hot data of each shard;

[0009] Determine the shard to which the hot data belongs through the first model, and move the hot data to the shard to which the hot data belongs; the first model is the first model trained by using the model training method provided in any embodiment of the present application.

[0010] The model training device provided by the embodiments of the present application includes:

[0011] Building unit: used to build a training set; the training set includes data in multiple shards; the dimensions of the data in the training set include one or more of the following dimensions: query frequency, write frequency, cognitive load, traceability analysis, co-occurrence times, user preferences, knowledge measurement metrics, abnormal requests;

[0012] Training unit: used to train a first model through the training set to obtain a trained first model; the first model is a fuzzy inference multi-classification model, and the first model is used to determine the shard to which the data input into the first model belongs.

[0013] The sharding device provided by an embodiment of the present application includes:

[0014] Monitoring unit: used to monitor each shard and identify the hot data of each shard;

[0015] Transmission unit: used to determine the shard to which the hot data belongs through the first model and move the hot data to the shard to which the hot data belongs; the first model is the first model trained by using the model training method described in any one of claims 1 to 6.

[0016] The electronic device provided by an embodiment of the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the model training method provided by any embodiment of the present application or the sharding method provided by any embodiment of the present application.

[0017] The chip provided by an embodiment of the present application includes: a processor, which is used to call and run a computer program from a memory, so that a device installed with the chip executes the model training method provided by any embodiment of the present application or the sharding method provided by any embodiment of the present application.

[0018] The storage medium provided by an embodiment of the present application is used to store a computer program, and the computer program enables a computer to execute the model training method provided by any embodiment of the present application or the sharding method provided by any embodiment of the present application.

[0019] The computer program product provided by an embodiment of the present application includes a computer program, and the computer program realizes the model training method provided by any embodiment of the present application or the sharding method provided by any embodiment of the present application when executed by a processor.

[0020] Through the model training method provided by the embodiments of the present application, a fuzzy inference multi-classification model is trained using data containing multiple shards, enabling the fuzzy inference multi-classification model to determine the shard to which the input data belongs. The fuzzy inference multi-classification model has a flexible data allocation strategy, optimizing the shard attribution of the data, thereby reducing unnecessary migration operations. Moreover, in the embodiments of the present application, multiple factors of the data are considered, including query frequency, write frequency, cognitive load, traceability analysis, co-occurrence times, user preferences, knowledge measurement indicators, abnormal requests, etc., enhancing the comprehensiveness of data evaluation and further improving the accuracy of sharding results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0022] Figure 1 is a flowchart of the model training method provided by the embodiments of the present application;

[0023] Figure 2 is a flowchart of the sharding method provided by the embodiments of the present application;

[0024] Figure 3 is a flowchart of the sharding transmission method provided by the embodiments of the present application;

[0025] Figure 4 is a schematic structural diagram of the model training device provided by the embodiments of the present application;

[0026] Figure 5 is a schematic structural diagram of the sharding device provided by the embodiments of the present application;

[0027] Figure 6 is a schematic structural diagram of the electronic device provided by the embodiments of the present application;

[0028] Figure 7 is a schematic structural diagram of the chip provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Next, the technical solutions in the embodiments of the present application will be described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0030] It should be noted that in the embodiments of the present application, the term "and / or" is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the embodiments of the present application, the character " / " generally represents an "or" relationship between the front and rear associated objects.

[0031] In the description of the embodiments of the present application, the term "corresponding" can represent a direct or indirect corresponding relationship between two parties, can also represent an association relationship between two parties, or can be a relationship such as indication and being indicated, configuration and being configured, etc.

[0032] To facilitate the understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all fall within the protection scope of the embodiments of the present application.

[0033] As the business scale of the database system expands, the requirements for the performance, capacity, data transmission efficiency, etc. of the database are continuously increasing. To address these challenges, sharding technology has become an important solution, which can improve the performance and scalability of the database. However, while adopting sharding technology, the problems of data transmission and synchronization become particularly critical. Existing MySQL sharding transmission technologies have some limitations in meeting real-time, reliability, and efficiency. During the data transmission process, problems such as network latency, load balancing, and data consistency need to be overcome. The related technologies determine whether the data thresholds stored in the loads of multiple database nodes are skewed, and according to the degree of skew of the data thresholds, new database sharding instances are created to provide storage space for the database node loads that need to be redistributed, further improving the performance of the database in large data volume scenarios.

[0034] However, the following disadvantages still exist in the related technologies: Real-time requirements: In some business scenarios, there are high real-time requirements for database operations, and a more rapid load balancing mechanism may be needed, which cannot be satisfied by the related technologies; Frequent sharding may lead to fragmentation: If the frequency of dynamic sharding is high, there may be a large number of small fragments in the database, which may increase the cost of queries, especially for queries that need to scan the entire database; Parameter selection: Unreasonable data dimension and threshold settings may lead to problems of over-sharding or under-sharding; Concurrency and transaction processing: The redistribution of database shards may have a certain impact on concurrency performance and transaction processing. When performing data migration, it may be necessary to consider how to ensure data consistency and transaction integrity; The introduction of dynamic sharding introduces complexity: The process of dynamically creating a database shard instance may introduce system dynamics and uncertainty, increasing the system complexity and management difficulty. The introduction of dynamic sharding is very likely to have a negative impact on the stability of the system; Data migration overhead: Creating a new database shard instance involves the migration of a large amount of data, which may introduce certain overheads, including network bandwidth, storage space, computing resources, etc. Especially in the scenario of a large amount of data, data migration may cause certain performance impacts and resource consumption.

[0035] Reference Figure 1 , Figure 1 is a schematic flowchart of the model training method provided by an embodiment of the present application. As Figure 1 shown, the model training method provided by this embodiment includes the following steps:

[0036] Step 101: Construct a training set; the training set includes data of multiple shards; the training set includes data of multiple shards; the dimensions of the data in the training set include one or more of the following dimensions: query frequency, write frequency, cognitive load, traceability analysis, co-occurrence times, user preference, knowledge measurement index, abnormal request.

[0037] In the embodiment of the present application, the query frequency represents the number of times the data is queried; the write frequency represents the number of times the data is written; the cognitive load represents whether the user's access to specific data will increase their cognitive load, and data with a high cognitive load may be more likely to become hot data; the traceability analysis represents the source and path of the data, and the location and connection of the data reflect its importance; the co-occurrence times represent whether the data often appears in the same query at the same time; the user preference represents the access patterns of different users; the knowledge measurement index represents whether the data contains innovative content and whether it is related to emerging technologies, concepts or research fields; the abnormal request represents the number of invalid or incorrect queries of the data.

[0038] In the embodiments of the present application, the data in the training set may be imbalanced data. At this time, oversampling operations can be performed on the minority-class shards in the training set to make the number of their data close to or the same as that of the majority-class shards, thereby improving the accuracy of the model.

[0039] Based on this, in an alternative embodiment of the present application, if the data in the training set is imbalanced data, oversampling processing is performed on the minority-class shards in the training set to obtain an oversampled training set.

[0040] In the embodiments of the present application, for all minority-class shards, samples in the minority-class dataset are sequentially selected and denoted as x i , and through the following formula:

[0041] Dist=||Φ(x i )-Φ(x)|| (1)

[0042] K nearest neighbors of x i in the feature space are obtained through formula (1), and all these K nearest neighbors belong to the minority class. K is a positive integer, and Φ(·) represents the mapping of the sample from the input space to the feature space. The function κ(·,·) is a kernel function. The kernel function can replace the inner product of two points in the high-dimensional feature space with the evaluation of a simple function of two patterns in the input space. Formula (1) can be simplified to formula (2).

[0043]

[0044] One of these K neighbors is randomly selected and denoted as x j , and according to the following formula:

[0045] x new =Φ(x i )+η×(Φ(x j )-Φ(x i )) (3)

[0046] a new sample is generated, where η is a random number between 0 and 1, and its role is to make the samples as diverse as possible. The categories of the newly generated samples are all minority classes. Then, according to the mapping x new of the new sample in the feature space, the preimage s new of the new sample in the input space is obtained.

[0047] In the embodiments of the present application, for each minority-class shard, one or more new samples are generated so that the number of data in the minority-class shard is close to or the same as the number of data in the majority-class shard.

[0048] Based on this, in an alternative embodiment of the present application, the oversampling processing of the minority-class shards in the training set includes:

[0049] For any minority-class slice, select a sample data from the minority-class slice, and determine K nearest neighbors of the sample data in the feature space; generate mappings of one or more new samples in the feature space according to the mapping of the sample data in the feature space and the K nearest neighbors; determine pre-images of the one or more new samples in the input space according to the mappings of the one or more new samples in the feature space.

[0050] In an embodiment of the present application, for a new sample, L nearest neighbors of the new sample in the feature space are determined through the K-nearest neighbor algorithm, and then the pre-image of the new sample in the input space, that is, the generated new data, is determined through the L nearest neighbors, where L is a positive integer.

[0051] Based on this, in an alternative embodiment of the present application, the determining the pre-images of the one or more new samples in the input space according to the mappings of the one or more new samples in the feature space includes:

[0052] For any new sample, determine L nearest neighbors of the mapping of the new sample in the feature space;

[0053] Determine the pre-image of the new sample in the input space according to the L nearest neighbors.

[0054] In an embodiment of the present application, use to represent the L minority-class nearest neighbors of the synthetic sample x new in the feature space, which can be obtained using the K-nearest neighbor algorithm Define the vector to represent the distance between the pre-image s new and in the input space, and use to represent the mean of x knn Create a matrix and the central matrix where E is the identity matrix and V is a K-dimensional column vector all of whose elements are 1. Then the matrix XC is a central matrix centered on . Perform singular value decomposition on XC, and the formula is as follows:

[0055]

[0056] where U 1 contains the left singular vectors of XC, U 2 usually represents additional orthogonal basis vectors, Σ is a diagonal matrix whose diagonal elements are non-negative real numbers, called singular values, and these singular values are arranged in descending order, reflecting the importance of the information in the original matrix XC; O represents the zero matrix, which is used here to fill the matrix to keep the structure complete; contains the right singular vectors of XC, is similar to U 2, representing additional orthogonal basis vectors. In practical applications, we usually only care about the main singular values and their corresponding singular vectors, so it can be simplified to the following formula:

[0057]

[0058] where U 1 is a matrix composed of a set of orthonormal column vectors, p is the projection on U 1 , so Define vectors and vector such that the following formula is satisfied:

[0059]

[0060] Since U 1 is a matrix composed of orthonormal column vectors, so the calculation formula of

[0061]

[0062] Sum the results of the above formula (7) from i = 1 to i = K. Since XC is the central matrix, the sum of the inner product terms is 0. Therefore, after summing, the following equation can be obtained:

[0063]

[0064] After transforming formula (8), the following equation is obtained:

[0065]

[0066] Substitute formula (9) into formula (7), and the following equation can be obtained:

[0067]

[0068] Formula (10) is expressed in matrix form as the following equation:

[0069]

[0070] Since is the central matrix, so Therefore, it can be obtained:

[0071]

[0072] Substitute formula (12) into formula (6) again, and the following can be obtained:

[0073]

[0074] The preimage s of the new sample in the input space can be obtained through formula (13). new The new sample and the original data together form a new training set, and the clustering centers are generated using the fuzzy inference multi-classification model on the training set. The new sample has the same dimension as the original data.

[0075] In the embodiments of the present application, the scores of each dimension index of the data in the training set can be normalized to between 0 and 1 to ensure the consistency of the dimensions of different dimension indexes, so as to facilitate the comprehensive calculation of the scores of hot data and identify hot data. The normalization process can use the min-max normalization method: For example, for the query frequency, the minimum query frequency and the maximum query frequency in the historical data can be taken for normalization processing.

[0076] Step 102: Train the first model through the training set to obtain the trained first model; the first model is a fuzzy inference multi-classification model, and the first model is used to determine the shard to which the data input into the first model belongs.

[0077] In the embodiments of the present application, the process of training the first model is to continuously iterate and update the parameters of the first model until a preset condition is reached; here, the parameters of the first model include the membership function and the clustering center, and the preset condition can be that the objective function reaches a preset threshold.

[0078] In the embodiments of the present application, the objective function of the multi-core fuzzy C-means clustering is as shown in the following formula:

[0079]

[0080] Among them, c is the number of clusters, and the number of clusters can be the same as the number of shards; μ ij represents the degree to which Φ(x j ) belongs to the i-th class, o i is the clustering center of the i-th class, and m is the fuzzy exponent, which controls the fuzziness of the clustering result.

[0081] According to the method of the ordinary fuzzy C-means algorithm, the membership degree in the feature space should satisfy the following formula:

[0082]

[0083] Among them, d is the Euclidean distance. For the distance calculation between the new sample and the original data in the input space, the following method can be adopted to establish the distance relationship between the feature space and the original space. For the Gaussian kernel function, for any sample x in the original minority-class dataset r The relationship between the distance between the synthetic sample in the feature space and the distance between x r and the preimage of the synthetic sample is:

[0084]

[0085] That is:

[0086]

[0087] Also, since Φ(s new ) is x new , the following can be obtained:

[0088]

[0089] Substituting formula (18) into formula (17), the distance D r in the input space between the new sample and x 2 (s new , x r ) can be obtained.

[0090] In the embodiments of the present application, the global kernel function focuses on the overall similarity of the data distribution, and the local kernel function focuses on processing the local features of the data. In order to combine the advantages of the global kernel function and the local kernel function, a multi-kernel function is constructed using Gaussian kernel functions and polynomial kernel functions with two different bandwidths, and the membership function is initialized. After multi-kernel mapping, the updated new cluster centers in the feature space are shown as follows:

[0091]

[0092] Then, the following can be calculated:

[0093]

[0094] Substituting formulas (20) and (21) into formula (15), the membership function can be updated as follows:

[0095]

[0096] By continuously iteratively updating the membership function and the cluster centers through the training set until the objective function J meets the preset threshold, the final cluster centers are obtained:

[0097]

[0098] In the embodiments of the present application, the membership function A i (x k ) in the antecedent of the rule satisfies the constraint conditions in the fuzzy C-means algorithm, that is After obtaining the cluster centers, the membership function in the antecedent of the rule is calculated according to the following formula:

[0099]

[0100] In the fuzzy inference multi-classification model, the output of an input vector is the weighted average of the outputs of each rule. Therefore, the total output can be expressed by the formula:

[0101]

[0102] where λ i0 , λ i1 , …, λ in are the linear parameters in the consequent of the rule, and the least squares method can be used to solve them and minimize the performance index.

[0103] Then, judge the shard to which the data belongs through the following formula:

[0104]

[0105] Exemplarily, if Y = 3.1, the shard to which the data belongs is the shard numbered 3; if Y = 1.9, the shard to which the data belongs is the shard numbered 2.

[0106] It should be noted that the value of n in the formula can be set according to the actual situation. Correspondingly, the value range of Y corresponding to different shards can also be adjusted according to the actual situation. For example, when there are two shards, it can be set as the following formula:

[0107]

[0108] Based on this, in an alternative embodiment of the present application, training the first model through the training set to obtain the trained first model includes:

[0109] Initializing the membership function of the first model through two Gaussian kernel functions and polynomial kernel functions with different bandwidths, and iteratively training the first model through the training set until a preset condition is met to obtain the trained first model; wherein, each training process of the iterative training includes:

[0110] Updating the parameters of the first model through the training set; the parameters of the first model include: membership function and clustering center.

[0111] Referring to Figure 2 , Figure 2 is the flowchart of the sharding method provided by the embodiment of the present application. As Figure 2 shown, the sharding method provided by this embodiment includes the following steps:

[0112] Step 201: Monitor each shard and identify the hot data of each shard.

[0113] In the embodiments of the present application, the identification of hot data is considered from multiple dimensions of data, including one or more of the following dimensions: query frequency, which represents the number of times the data is queried; write frequency, which represents the number of times the data is written; cognitive load, which represents whether the user's access to specific data will increase their cognitive load, and data with a high cognitive load may be more likely to become hot data; traceability analysis, which represents the source and path of the data, and the location and connection of the data reflect its importance; co-occurrence times, which represents whether the data often appears in the same query at the same time; user preference, which represents the access patterns of different users; knowledge measurement index, which represents whether the data contains innovative content and whether it is related to emerging technologies, concepts or research fields; abnormal requests represent the number of invalid or incorrect queries of the data.

[0114] Each index is quantified, then scored and standardized. The scores of each index are between 0 and 1. By presetting the weights of each index, the total score of each index of the data is calculated. When it exceeds the preset score threshold, the data is determined to be hot data.

[0115] Based on this, in an alternative embodiment of the present application, the identification of the hot data of each shard includes:

[0116] According to the first information of the data, it is determined whether the data is hot data; the first information includes one or more of the following information: query frequency, write frequency, cognitive load, traceability analysis, co-occurrence times, user preference, knowledge measurement index, abnormal requests.

[0117] Exemplarily, taking the four dimensions of query frequency, write frequency, co-occurrence times, and abnormal request times as examples, the calculation methods of other indexes are similar. Suppose the user order data is as shown in Table 1:

[0118]

[0119] Table 1

[0120] Using min-max normalization, the query frequency of order 1 is calculated as: (500 - 300) / (1500 - 300) = 0.17; order 2: (300 - 300) / (1500 - 300) = 0; order 3: (1500 - 300) / (1500 - 300) = 1; order 4: (800 - 300) / (1500 - 300) = 0.42. The other indexes (write frequency, co-occurrence times, abnormal request times) are standardized in the same way.

[0121] Then set the weight of each index: query frequency: 0.5, write frequency: 0.3, co-occurrence times: 0.1, abnormal request times: 0.1. The comprehensive score is the sum of the values of each index after standardization multiplied by the threshold.

[0122] Order 1: 0.5 * 0.17 + 0.3 * 0.4 + 0.1 * 0.25 + 0.1 * 0 = 0.218;

[0123] Order 2: 0.5 * 0 + 0.3 * 1 + 0.1 * 0 + 0.1 * 0.5 = 0.38;

[0124] Order 3: 0.5 * 1 + 0.3 * 0.2 + 0.1 * 1 + 0.1 * 0 = 0.64;

[0125] Order 4: 0.5 * 0.42 + 0.3 * 0.3 + 0.1 * 0.5 + 0.1 * 1 = 0.475.

[0126] Set a scoring threshold for hotspot data. For example, if the comprehensive score > 0.5, then according to the calculation, Order 3 is hotspot data.

[0127] In the embodiments of the present application, it is also possible to shard the data sources that need to be sharded to obtain multiple shards, and then monitor each of the multiple shards.

[0128] In the embodiments of the present application, the database sharding and table partitioning method can adopt horizontal sharding and / or vertical sharding. Horizontal sharding is applicable to the situation where the data scale is large and can be horizontally extended. The horizontal sharding method is relatively more flexible. Each shard is only responsible for part of the data, thus supporting larger-scale data storage and query, and does not require changing the table structure according to the business. For the sharding rules, one of the following rules can be adopted: sharding by range, sharding by hash, sharding by modulus, sharding by user ID, sharding by hash slot, etc. Exemplarily, considering the distribution uniformity of the system, the sharding rule of sharding by hash can be selected to reduce the problem that hotspot data is concentrated on a certain shard.

[0129] Step 202: Determine the shard to which the hotspot data belongs through the first model, and move the hotspot data to the shard to which the hotspot data belongs; the first model is the first model trained by using the model training method provided in any embodiment of the present application.

[0130] In the embodiments of the present application, a fuzzy inference multi-classification model is used for calculation. The hotspot data indicators are not linearly separable. The fuzzy inference multi-classification model provided in the embodiments of the present application adds a kernel function to map the samples to a high-dimensional space to make them linearly separable, and a multi-core processor can process different iterative steps in parallel, accelerating the whole process and can also better process large-scale data sets.

[0131] In the embodiments of the present application, it is also possible to listen and receive the transaction information of multiple shards through a transaction coordinator, identify the heat of incremental data, and perform balanced storage on potential heat data.

[0132] Based on this, in an optional implementation manner of this application, it further includes: monitoring the transaction information of each shard to obtain incremental data;

[0133] If the incremental data is hot data, determine the shard of the incremental data through the first model, and move the incremental data to the shard to which the incremental data belongs.

[0134] Reference Figure 3 , Figure 3 is a schematic flowchart of the shard transmission method provided by the embodiment of this application. As Figure 3 shown, in this embodiment, taking the MySQL database as an example, the design of the MySQL shard transmission component involves multiple key components, including a message broker, a task coordinator, a database transmission service (Data Transmission Service, DTS), and a transaction coordinator. The server obtains the data source that needs to be sharded; sends the data source to the message broker to perform the operation of saving the message and publishing it to the queue, and outputs the asynchronous synchronization queue message to the task coordinator; the task coordinator receives the asynchronous synchronization queue message from the message broker, performs traffic balancing processing, outputs the balanced data to the database transmission service, and continuously monitors each shard after sharding to identify the hot data of each shard; determines the shard to which the hot data belongs through the first model, and moves the hot data to the shard to which the hot data belongs; the first model is the first model trained by using the model training method provided in any embodiment of this application; the database transmission service receives the data from the task coordinator and performs shard transmission operations; the transaction coordinator listens to and receives the transaction information of multiple shards, identifies the heat of the incremental data, and evenly stores the potential heat data through the shard method provided by the embodiment of this application.

[0135] Through the collaborative work of these components, the MySQL shard transmission component realizes efficient, scalable, and reliable data transmission. The message broker provides an asynchronous and decoupled communication mechanism, shards by using the shard method provided by the embodiment of this application, and screens and balances the hot data for the newly generated incremental data after sharding. The mechanism for automatically adjusting the balance and hot data ensures that the system can automatically adapt to load changes during operation, improving the stability and performance of the system.

[0136] The sharding method provided by the embodiments of the present application can perform database sharding for real-time task coordination, improve the real-time performance of services, and does not require strict parameter selection, avoiding the data balance tilt that may be caused by unreasonable parameters. It also averages the hot query data of the database, avoiding the gap in query performance and time when querying sharded data. Moreover, using a transaction coordinator to ensure the consistency of each shard helps to meet the scalability requirements of the system; continuously monitor the access times of each piece of data. Based on these real-time data access statistics, the position of the data between different shards can be dynamically reassigned to ensure that the total number of accesses on each shard can remain balanced during the operation of the system. This feature innovates on real-time data access patterns and feedback mechanisms to ensure the immediate adjustment of data distribution in a distributed system; utilize a fuzzy inference multi-classification model to flexibly handle the data allocation strategy. Through the quantified membership degree and fuzzy inference mechanism, the sharding attribution of the data is optimized, thereby reducing unnecessary migration operations and reducing the number of times the data moves between shards. The fuzzy inference multi-classification model is more flexible than the traditional hard classification model. When considering hot data, this flexibility can better adapt to different types of data distributions and access patterns. The output of the fuzzy inference multi-classification model can be obtained through fuzzy inference, which provides the interpretability of the membership degree of each data point on each shard, helps to understand the data distribution and adjust the model parameters, and adds multi-core fuzzy clustering as an innovation to make full use of the parallelism of multi-core processors to improve the computing efficiency; considering the dynamic nature of the access pattern, the method provided by the embodiments of the present application is adaptable and can flexibly adjust the position of the data in the shard according to the change of the data access pattern. By continuously monitoring and analyzing the change trend of the data access pattern, the data distribution can be adjusted in a timely manner to ensure that the shards remain in a balanced state. This feature ensures the high efficiency and flexibility of the system in the face of changing workloads and access patterns.

[0137] The embodiments of the present application also provide a model training device. Refer to Figure 4 , Figure 4 which is a schematic structural diagram of the model training device provided by the embodiments of the present application. The model training device in this embodiment includes:

[0138] A construction unit: used to construct a training set; the training set includes data from multiple shards; the dimensions of the data in the training set include one or more of the following dimensions: query frequency, write frequency, cognitive load, traceability analysis, co-occurrence times, user preferences, knowledge measurement indicators, and abnormal requests;

[0139] A training unit: used to train a first model through the training set to obtain a trained first model; the first model is a fuzzy inference multi-classification model, and the first model is used to determine the shard to which the input data belongs.

[0140] In the embodiment of the present application, the construction unit is used to perform oversampling processing on the minority class shards in the training set if the data in the training set is imbalanced data, so as to obtain an oversampled training set.

[0141] In the embodiment of the present application, for any minority class shard, the construction unit is used to select a sample data from the minority class shard, determine K nearest neighbors of the sample data in the feature space; generate mappings of one or more new samples in the feature space according to the mapping of the sample data in the feature space and the K nearest neighbors; determine pre-images of the one or more new samples in the input space according to the mappings of the one or more new samples in the feature space.

[0142] In the embodiment of the present application, for any new sample, the construction unit is used to determine L nearest neighbors of the mapping of the new sample in the feature space; determine the pre-image of the new sample in the input space according to the L nearest neighbors.

[0143] In the embodiment of the present application, the training unit is used to initialize the membership function of the first model through two Gaussian kernel functions and polynomial kernel functions with different bandwidths, and perform iterative training on the first model through the training set until a preset condition is met, so as to obtain a trained first model; wherein, each training process of the iterative training includes: updating the parameters of the first model through the training set; the parameters of the first model include: the membership function and the cluster center.

[0144] Those skilled in the art should understand that Figure 4 The implementation functions of the units in the model training device shown can be understood with reference to the relevant descriptions of the foregoing method. Figure 4 The functions of the units in the model training device shown can be realized by a program running on a processor, or can be realized by specific logic circuits.

[0145] The embodiment of the present application further provides a sharding device, refer to Figure 5 , Figure 5 is a schematic structural diagram of the sharding device provided by the embodiment of the present application. The sharding device in this embodiment includes:

[0146] Monitoring unit: used to monitor each shard and identify hot data of each shard;

[0147] Transmission unit: used to determine the shard to which the hot data belongs through the first model, and move the hot data to the shard to which the hot data belongs; the first model is the first model trained by using the model training method provided in any embodiment of the present application.

[0148] In the embodiments of the present application, the monitoring unit is configured to determine whether the data is hot data according to the first information of the data; the first information includes one or more of the following information: query frequency, write frequency, cognitive load, traceability analysis, co-occurrence times, user preferences, knowledge measurement indicators, and abnormal requests.

[0149] In the embodiments of the present application, the monitoring unit is configured to monitor the transaction information of each shard to obtain incremental data; the transmission unit: if the incremental data is hot data, then determine the shard of the incremental data through the first model, and move the incremental data to the shard to which the incremental data belongs.

[0150] Those skilled in the art should understand, Figure 5 The implementation functions of the units in the sharding device shown can be understood with reference to the relevant descriptions of the foregoing method. Figure 5 The functions of the units in the sharding device shown can be implemented by a program running on a processor or by specific logic circuits.

[0151] Figure 6 It is a schematic structural diagram of an electronic device 600 provided by the embodiments of the present application. Figure 6 The electronic device 600 shown includes a processor 610. The processor 610 can call and run a computer program from a memory to implement the method in the embodiments of the present application.

[0152] Optionally, as Figure 6 shown, the electronic device 600 may further include a memory 620. Among them, the processor 610 can call and run a computer program from the memory 620 to implement the method in the embodiments of the present application.

[0153] Among them, the memory 620 can be an independent device from the processor 610 or can be integrated in the processor 610.

[0154] Optionally, as Figure 6 shown, the electronic device 600 may further include a transceiver 630. The processor 610 can control the transceiver 630 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0155] Among them, the transceiver 630 can include a transmitter and a receiver. The transceiver 630 may further include an antenna, and the number of antennas can be one or more.

[0156] The electronic device 600 may specifically be the model training device / sharding device of the embodiments of the present application, and the electronic device 600 may implement the corresponding processes implemented by the model training device / sharding device in each method of the embodiments of the present application. For the sake of brevity, details are not described herein again.

[0157] Exemplarily, an embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 610 of the communication device 600 to complete the steps of any of the foregoing methods.

[0158] Figure 7 It is a schematic structural diagram of the chip of the embodiment of the present application. Figure 7 The shown chip 700 includes a processor 710. The processor 710 can call and run a computer program from the memory to implement the methods in the embodiments of the present application.

[0159] Optionally, as Figure 7 shown, the chip 700 may further include a memory 720. Among them, the processor 710 can call and run a computer program from the memory 720 to implement the methods in the embodiments of the present application.

[0160] Among them, the memory 720 may be a separate device independent of the processor 710 or may be integrated in the processor 710.

[0161] Optionally, the chip 700 may further include an input interface 730. Among them, the processor 710 can control the input interface 730 to communicate with other devices or chips. Specifically, it can obtain information or data sent by other devices or chips.

[0162] Optionally, the chip 700 may further include an output interface 740. Among them, the processor 710 can control the output interface 740 to communicate with other devices or chips. Specifically, it can output information or data to other devices or chips.

[0163] This chip can be applied to the electronic device 600 in the embodiments of the present application, and this chip can implement the corresponding processes implemented by the electronic device 600 in each method of the embodiments of the present application. For the sake of brevity, details are not described herein again.

[0164] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0165] It should be understood that the processor in the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software units in the decoding processor. The software unit may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0166] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.

[0167] It should be understood that the above memory is by way of example but not limitation. For example, the memory in the embodiments of the present application can also be a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synch link DRAM (SLDRAM), and a direct rambus random access memory (DR RAM), and so on. That is to say, the memory in the embodiments of the present application is intended to include but not be limited to these and any other suitable types of memory.

[0168] An embodiment of the present application also provides a storage medium for storing a computer program. The storage medium can be applied to the electronic device 600 in the embodiment of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the electronic device 600 in each method of the embodiment of the present application. For the sake of brevity, details are not described herein again.

[0169] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0170] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and details are not described herein again.

[0171] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical, or other form.

[0172] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0174] When the above-described function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or an electronic device 600, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0175] As described above, the above are only the specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Constructing a training set; the training set includes data of multiple shards; The dimensions of the data in the training set include one or more of the following dimensions: query frequency, write frequency, cognitive load, source tracing analysis, co-occurrence times, user preferences, knowledge measurement indicators, and abnormal requests; Train the first model using the training set to obtain a trained first model; The first model is a fuzzy reasoning multi-classification model, and the first model is used to determine the shard to which the data input into the first model belongs.

2. The method according to claim 1, characterized in that Also includes: If the data in the training set is unbalanced data, oversampling is performed on the minority class slices in the training set to obtain an oversampled training set.

3. The method according to claim 2, characterized in that The oversampling of the minority class slices in the training set includes: For any minority class slice, select a sample data from the minority class slice and determine the K nearest neighbors of the sample data in the feature space; generate the mapping of one or more new samples in the feature space according to the mapping of the sample data in the feature space and the K nearest neighbors; determine the original image of the one or more new samples in the input space according to the mapping of the one or more new samples in the feature space.

4. The method according to claim 3, characterized in that The determining, according to the mapping of the one or more new samples in the feature space, the original image of the one or more new samples in the input space comprises: For any new sample, determine the L nearest neighbors of the mapping of the new sample in the feature space; An original image of the new sample in the input space is determined according to the L nearest neighbors.

5. The method according to any one of claims 1 to 4, characterized in that Training the first model by using the training set to obtain the trained first model includes: The membership function of the first model is initialized by using two Gaussian kernel functions and a polynomial kernel function with different bandwidths, and the first model is iteratively trained by using the training set until a preset condition is met to obtain a trained first model; wherein each training process of the iterative training includes: The parameters of the first model are updated through the training set; the parameters of the first model include: membership function and cluster center.

6. A sharding method, characterized in that: include: Monitor each shard and identify hotspot data of each shard; The shard to which the hot data belongs is determined through a first model, and the hot data is moved to the shard to which the hot data belongs; the first model is a first model trained using the model training method described in any one of claims 1 to 6.

7. The method according to claim 6, characterized in that The identifying hotspot data of each shard includes: Determine whether the data is hot data based on first information of the data; the first information includes one or more of the following information: query frequency, write frequency, cognitive load, source tracing analysis, co-occurrence times, user preferences, knowledge measurement indicators, and abnormal requests.

8. The method according to claim 6 or 7, characterized in that: Also includes: Monitor the transaction information of each shard to obtain incremental data; If the incremental data is hot data, the shard of the incremental data is determined by the first model, and the incremental data is moved to the shard to which the incremental data belongs.

9. A model training device, characterized in that: include: Construction unit: used to construct a training set; the training set includes data of multiple shards; The dimensions of the data in the training set include one or more of the following dimensions: query frequency, write frequency, cognitive load, source tracing analysis, co-occurrence times, user preferences, knowledge measurement indicators, and abnormal requests; Training unit: used for training the first model through the training set to obtain the trained first model; The first model is a fuzzy reasoning multi-classification model, and the first model is used to determine the shard to which the data input into the first model belongs.

10. A slicing device, characterized in that: include: Monitoring unit: used to monitor each shard and identify hotspot data of each shard; Transmission unit: used to determine the shard to which the hotspot data belongs through a first model, and move the hotspot data to the shard to which the hotspot data belongs; the first model is a first model trained using the model training method described in any one of claims 1 to 6.

11. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the model training method as described in any one of claims 1 to 5, or the sharding method as described in any one of claims 6 to 8.

12. A chip, characterized in that: include: A processor, used to call and run a computer program from a memory, so that a device equipped with the chip executes the model training method described in any one of claims 1 to 5, or the sharding method described in any one of claims 6 to 8.

13. A storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the model training method as described in any one of claims 1 to 5, or the sharding method as described in any one of claims 6 to 8.

14. A computer program product comprising a computer program, characterized in that When executed by a processor, the computer program implements the model training method as described in any one of claims 1 to 5, or the sharding method as described in any one of claims 6 to 8.

Images