Training method, device, equipment and medium for memory detection model

Abstract

The embodiments of the present disclosure relate to a method, apparatus, device, and medium for training a memory detection model. The determination method includes: obtaining raw data statistics of account data stored in at least three memory storage systems; obtaining the actual space occupancy of the memory space in each of the memory storage systems; using the raw data statistics and actual space occupancy corresponding to each of the memory storage systems as training samples, and training and learning the parameters of a set memory detection model to determine all parameters in the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the raw data statistics of the account data stored in the memory storage system. The embodiments of the present disclosure can obtain a memory detection model by using the raw data statistics and actual space occupancy in the memory storage system to fit and learn the memory detection model, and the model has universal applicability.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer data storage technology, and in particular to a method, apparatus, device, and medium for training a memory detection model. Background Art

[0002] In computer technology, memory is an essential resource for program execution. This is particularly true for internet-based services, where servers need to cache large amounts of data in memory to quickly support data responses. Redis, a high-performance in-memory key-value database, is gaining increasing adoption in various fields requiring high-performance storage, particularly internet companies.

[0003] In the actual application of Redis, it is often necessary to store data of multiple accounts in the same Redis cluster. At this time, being able to accurately count the memory space occupied by each account becomes a necessary condition for fine-grained resource management.

[0004] Related technologies require a detailed analysis of the Redis memory model, such as the memory usage rules for various supported data types and the rules for generating additional memory overhead. Therefore, once the number of stored account data items, data types, and data sizes are known, Redis's memory usage can be determined based on precise calculation rules.

[0005] However, the problem with these technologies is that their memory usage calculation methods are heavily dependent on the Redis memory model and operating environment. Changes to the Redis version, or any changes to the CPU, operating system, or compiler version in the operating environment, can cause the calculation rules to change, resulting in inaccurate memory usage results. This requires re-analyzing the memory model to determine the most appropriate calculation rules for the current situation. This approach clearly cannot adapt to diverse and complex application scenarios, nor can it accommodate rapid upgrades. Summary of the Invention

[0006] The present disclosure provides a method, apparatus, device, and medium for training a memory detection model to at least address the problem in related art where memory space occupancy detection methods strictly rely on memory models. The technical solutions of the present disclosure are as follows:

[0007] According to a first aspect of an embodiment of the present disclosure, a method for training a memory detection model is provided, comprising:

[0008] Obtaining raw data statistics of account data stored in at least three memory storage systems, where the raw data of the account data is data directly stored and accessed in memory according to business needs of the account, and the raw data statistics of the account data is the storage byte length of the raw data;

[0009] Obtaining the actual space occupancy of the memory space in each of the memory storage systems, where the actual space occupancy is the size of the memory space actually occupied in the memory storage system;

[0010] The raw data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the parameters of the set memory detection model are trained and learned to determine all parameters in the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the raw data statistics of the account data stored in the memory storage system.

[0011] Optionally, if the memory detection model is a linear function, the raw data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples to train and learn the parameters of the set memory detection model, including:

[0012] The original data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the least squares method is used to fit the parameters of the set memory detection model so that the residual function of the fitting result converges.

[0013] Optionally, if the memory storage system is a key-value pair storage system, the formula of the memory detection model is as follows:

[0014] Detection memory usage = a * number of key-value pairs + b * amount of data stored in the key-value pairs + c * number of storage nodes;

[0015] Among them, a, b, and c are the parameters of the memory detection model.

[0016] Optionally, obtaining raw data statistics of account data stored in at least three memory storage systems includes:

[0017] For each memory storage system, traverse and access any storage node of a memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the storage node; each memory storage cluster includes at least two storage nodes, and the memory storage cluster adopts a node space balancing rule;

[0018] The raw data statistic of the memory storage cluster is determined according to the raw data statistic of a storage node and the number of storage nodes in the memory storage cluster.

[0019] Optionally, obtaining raw data statistics of account data stored in at least three memory storage systems includes:

[0020] For each memory storage system, a traversal access is performed on the backup storage nodes in the memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the backup storage nodes; wherein the memory storage cluster includes at least one node shard, each node shard includes a primary storage node and at least one backup storage node, the primary storage node is used to provide online services, and the backup storage node is used to synchronize data of the primary storage node for backup;

[0021] The original data statistics of the memory storage cluster are determined according to the original data statistics of the backup storage node.

[0022] Optionally, before obtaining the raw data statistics of the account data stored in the at least three memory storage systems, the method further includes:

[0023] The cluster name and the number of storage nodes of each memory storage cluster are obtained by scanning the database meta information of each memory storage system.

[0024] Optionally, after the step of determining all parameters in the memory detection model, the method further includes:

[0025] When it is determined that a storage memory model change event occurs, the steps of obtaining the original data statistics and the actual space occupancy and training to determine the parameters in the memory detection model are triggered to be re-executed.

[0026] Optionally, when determining that a storage memory model change event occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training to determine parameters in the memory detection model includes at least one of the following steps:

[0027] When it is determined that a version change of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0028] When determining that a change in the compilation parameters of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0029] When it is determined that a change in the operating parameters of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0030] When it is determined that a change occurs in the operating environment of the memory storage system, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0031] When it is determined that a change in the data feature distribution of the account data stored in the memory storage system occurs, the steps of obtaining the original data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model are triggered, wherein the data features include the data type and / or the amount of a single data item.

[0032] According to a second aspect of an embodiment of the present disclosure, a training device for a memory detection model is provided, comprising:

[0033] a statistical value acquisition unit configured to acquire raw data statistical values ​​of account data stored in at least three memory storage systems, wherein the raw data of the account data is data directly stored and accessed in the memory according to business needs of the account, and the raw data statistical value of the account data is the storage byte length of the raw data;

[0034] an actual space occupancy acquisition unit configured to acquire an actual space occupancy of the memory space in each of the memory storage systems, wherein the actual space occupancy is the size of the memory space actually occupied in the memory storage system;

[0035] The model training unit is configured to use the original data statistics and actual space occupancy corresponding to each of the memory storage systems as training samples, and train and learn the parameters of the set memory detection model to determine all parameters in the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the original data statistics of the account data stored in the memory storage system.

[0036] Optionally, if the memory detection model is a linear function, the model training unit is configured to:

[0037] The original data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the least squares method is used to fit the parameters of the set memory detection model so that the residual function of the fitting result converges.

[0038] Optionally, if the memory storage system is a key-value pair storage system, the formula of the memory detection model is as follows:

[0039] Detection memory usage = a * number of key-value pairs + b * amount of data stored in the key-value pairs + c * number of storage nodes;

[0040] Among them, a, b, and c are the parameters of the memory detection model.

[0041] Optionally, the statistical value acquisition unit includes:

[0042] The traversal access subunit is configured to, for each memory storage system, perform a traversal access on any storage node of a memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the storage node; each memory storage cluster includes at least two storage nodes, and the memory storage cluster adopts a node space balancing occupancy rule;

[0043] The first data determination subunit is configured to determine the raw data statistic value of the memory storage cluster according to the raw data statistic value of a storage node and the number of storage nodes in the memory storage cluster.

[0044] Optionally, the statistical value acquisition unit includes:

[0045] A backup node scanning subunit is configured to, for each memory storage system, perform traversal access to backup storage nodes in a memory storage cluster serving as the memory storage system to determine raw data statistics of account data in the backup storage nodes; wherein the memory storage cluster includes at least one node shard, each node shard includes a primary storage node and at least one backup storage node, the primary storage node is used to provide online services, and the backup storage node is used to synchronize data of the primary storage node for backup;

[0046] The second data determination subunit is configured to determine the original data statistical value of the memory storage cluster according to the original data statistical value of the backup storage node.

[0047] Optionally, the method further comprises:

[0048] The meta information scanning unit is configured to obtain the cluster name and the number of storage nodes of each of the memory storage clusters by scanning the database meta information of each of the memory storage systems.

[0049] Optionally, also include:

[0050] The function update unit is configured to trigger the steps of re-executing the steps of obtaining the original data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model when determining that a storage memory model change event occurs.

[0051] Optionally, the function updating unit includes:

[0052] A first function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining parameters in the memory detection model when determining that a version change of the memory storage system occurs;

[0053] A second function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters of the memory detection model when determining that a compilation parameter change of the memory storage system occurs;

[0054] A third function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters of the memory detection model when determining that a change in the operating parameters of the memory storage system occurs;

[0055] a fourth function updating subunit configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining parameters in the memory detection model when determining that a change in the operating environment of the memory storage system occurs;

[0056] The fifth function update sub-unit is configured to trigger the steps of re-executing the steps of obtaining the original data statistics and the actual space occupancy, and training to determine the parameters in the memory detection model when it is determined that the data feature distribution of the account data stored in the memory storage system has changed, wherein the data features include the data type and / or the amount of a single data item.

[0057] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0058] processor;

[0059] a memory for storing instructions executable by the processor;

[0060] The processor is configured to execute the instructions to implement the training method of the memory detection model provided in the embodiment of the first aspect of the present disclosure.

[0061] According to the fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the memory detection model provided in the embodiment of the first aspect of the present disclosure.

[0062] According to the fifth aspect of the embodiments of the present disclosure, a computer program product is provided for use in combination with an electronic device, wherein the computer program product includes a computer-readable storage medium and a computer program mechanism embedded therein, and after the program is loaded into a computer and executed, the training method of the memory detection model provided in the embodiment of the first aspect of the present disclosure can be implemented.

[0063] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0064] A memory detection model can be obtained by fitting and learning the memory detection model using raw data statistics and actual space usage in the memory storage system. This method does not rely on analyzing the memory model of the memory storage system. It does not strictly depend on the memory storage system version, operating environment, or account data, making it universally applicable. When it is necessary to understand the space usage of the memory storage system, the raw data statistics of the account data can quickly and accurately detect the actual space usage, providing guidance for memory space maintenance and management. Memory space maintenance and management can include resource statistics, budgeting, capacity quotas, and capacity billing.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0067] Figure 1A The figure is a flowchart of a method for training a memory detection model according to an exemplary embodiment.

[0068] Figure 1B This is a schematic diagram of the architecture of a memory storage cluster applicable to an embodiment of the present disclosure.

[0069] Figure 2 It is a flowchart of a method for training a memory detection model shown in an exemplary embodiment.

[0070] Figure 3 The figure is a block diagram of a training device for a memory detection model according to an exemplary embodiment.

[0071] Figure 4 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] In order to enable ordinary people in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0073] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0074] Figure 1A This is a flowchart illustrating a method for training a memory detection model according to an exemplary embodiment. This embodiment is applicable to training memory detection models, particularly for training memory detection models in large-scale memory storage systems that support internet business services. This method can be performed by a memory detection model training device, which can be implemented in hardware and / or software and can be configured in a device in the memory storage system or in a third-party device independent of the memory storage system.

[0075] In the embodiment of the present disclosure, the memory storage system is described as a memory storage cluster. Figure 1B As shown, a memory storage cluster typically includes at least one node shard 102, each of which can independently provide memory for data storage. A memory storage cluster typically includes proxy middleware (or shard proxy, nutcracker) 101, which distributes data access requests received from clients to each node shard 102 via proxy middleware 101 to respond to data access requests. Each node shard 102 may include a storage node. Alternatively, typically for data security, a node shard 102 includes a primary storage node (master) 1021 and at least one backup storage node (slave) 1022. The primary storage node 1021 is used to provide online services, and the backup storage node 1022 is used to synchronize data from the primary storage node 1021 for backup. For example, when the memory storage cluster supports a certain internet service, the client accesses the server and pulls data from the server. The primary storage node of the memory storage cluster supports the client's online data access; the backup storage node obtains data from the primary storage node for backup according to the synchronization strategy, so that it can switch to support online services in the event of a primary storage node failure. In-memory storage clusters can be implemented using various memory protocols. For example, Redis in-memory storage clusters are widely used on the internet. For internet businesses, one or more in-memory storage clusters may be set up to support data access.

[0076] For in-memory storage clusters, the primary memory usage is account data, such as business data that clients need to retrieve when accessing servers. In addition to the raw data statistics required to store account data, additional memory overhead is also required for data storage and memory management. Once the raw data statistics are known, the memory overhead needs to be measured. Different account data, memory storage clusters, or operating environments can cause memory overhead to vary, necessitating the training of an effective memory detection model to monitor memory usage for effective memory management.

[0077] like Figure 1A As shown, the training method of the memory detection model provided in the embodiment of the present disclosure includes:

[0078] In step S110, raw data statistics of account data stored in at least three memory storage systems are obtained, where the raw data of the account data is data directly stored and accessed in the memory according to business needs, and the raw data statistics of the account data is the storage byte length of the raw data.

[0079] In this embodiment, operations are performed on at least three memory storage systems to obtain raw data statistics for the account data stored in each memory storage system. The raw data of the account data is the data directly stored and accessed in memory based on business needs. The raw data statistics for the account data are the byte length of the raw data stored. For example, in Redis, raw account data is stored as key-value pairs. The raw data statistics are the byte lengths stored in the key and value fields.

[0080] In step S120, the actual space occupancy of the memory space in each of the memory storage systems is obtained, where the actual space occupancy is the size of the memory space actually occupied in the memory storage system.

[0081] In this embodiment, the actual space occupied is the actual memory space occupied in the memory storage system. Due to the existence of additional memory overhead, the actual memory space occupied is larger than the original data volume.

[0082] The raw data statistics described above can be obtained by scanning the in-memory storage system. For example, the Spark RedisScanner scanner in a Redis cluster can traverse any storage node in the in-memory storage cluster to directly obtain the number of key-value pairs and the amount of data stored in each key-value pair (including the length of the key and value strings) as raw data statistics. Spark-based RedisScanner's parallel traversal access can significantly improve the efficiency of Redis capacity information collection.

[0083] The actual memory usage can be obtained by accessing the cluster management platform and querying it. The cluster management platform stores the actual memory usage of each memory storage system. For example, the actual memory usage of each Redis cluster can be obtained by accessing the Redis management platform's interface.

[0084] In step S130, the original data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the parameters of the set memory detection model are trained and learned to determine all parameters in the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the original data statistics of the account data stored in the memory storage system.

[0085] In this embodiment, the memory detection model is pre-set and can be set as a function according to needs or experience. The memory detection model can express the relationship between the original data statistics and the actual space occupancy. Optionally, the memory detection model can be a linear function, a nonlinear function or a machine learning model. There are many ways to use training samples to learn the parameters in the memory detection model, such as least squares fitting, conventional equation solving method, and other methods based on machine learning or numerical optimization, such as multi-island genetic algorithm. The original data statistics and actual space occupancy corresponding to multiple memory storage systems are used as multiple training samples, and the memory detection model can be trained and learned to determine the parameters of the memory detection model.

[0086] The technical solution of the disclosed embodiment can determine the memory detection model by learning the memory detection model using the raw data statistics and actual space occupancy in the memory storage system. This method is not determined by analyzing the memory model of the memory storage system. There is no strict dependency on the version of the memory storage system itself, the operating environment, and the account data, and it has universal applicability. When it is necessary to understand the space occupancy of the memory storage system, the actual space occupancy can be quickly and accurately detected based on the raw data statistics of the account data, which has guiding significance for the maintenance and management of the memory space.

[0087] On the basis of the above-mentioned embodiments of the present disclosure, optionally, the original data statistics can be estimated based on the node space balancing occupancy rule adopted by the memory storage system, thereby further saving the resources and time consumed by the scanning task. The node space balancing occupancy rule can be balanced according to the hash of the key field (Key) of the data, and the hash is used to ensure that the data is basically evenly stored in each main storage node. For example, in a Redis cluster, since the proxy middleware or proxy node (nutcreacker) that manages the memory storage cluster will ensure that the account data is basically evenly stored in each main storage node through hashing, the data storage situation of other node shards can be estimated by scanning the storage nodes of a node shard.

[0088] Specifically, obtaining raw data statistics of account data stored in at least three memory storage systems may include:

[0089] For each memory storage system, traverse and access any storage node of a memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the storage node; each memory storage cluster includes at least two storage nodes, and the memory storage cluster adopts a node space balancing rule;

[0090] The raw data statistic of the memory storage cluster is determined according to the raw data statistic of a storage node and the number of storage nodes in the memory storage cluster.

[0091] In the above solution, after scanning a storage node, the raw data statistics in that storage node are obtained. Then, multiplying the raw data statistics by the number of storage nodes, the raw data statistics for the entire in-memory storage cluster can be estimated. It is best to scan the backup storage nodes in the node shard to avoid the scanning process affecting the performance of the primary storage nodes that support online data access.

[0092] For a node shard with active and standby nodes, optionally, the step of obtaining raw data statistics of account data stored in at least three memory storage systems includes:

[0093] For each memory storage system, a traversal access is performed on the backup storage nodes in the memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the backup storage nodes; wherein the memory storage cluster includes at least one node shard, each node shard includes a primary storage node and at least one backup storage node, the primary storage node is used to provide online services, and the backup storage node is used to synchronize data of the primary storage node for backup;

[0094] The original data statistics of the memory storage cluster are determined according to the original data statistics of the backup storage node.

[0095] In the above solution, because the data in the backup storage nodes and the primary storage nodes is backed up synchronously, the scan results are consistent. You can scan all backup storage nodes separately, or just one or a small number of backup storage nodes, and then estimate the raw data statistics of the entire memory storage cluster based on the number of backup storage nodes.

[0096] Optionally, before obtaining the raw data statistics of the account data stored in the at least three memory storage systems, the method further includes:

[0097] The cluster name and the number of storage nodes of each memory storage cluster are obtained by scanning the database meta information of each memory storage system.

[0098] Taking the Redis cluster as an example, you can first query the MySQL database metadata to obtain the cluster name and number of storage nodes of the Redis cluster.

[0099] In the above scheme, the cluster name and storage node number of the memory storage cluster of each memory storage system can be obtained in advance, so that the storage nodes of each memory storage cluster can be traversed and accessed according to the cluster name and storage node number of each memory storage cluster.

[0100] Figure 2 This is a flowchart of a method for training a memory detection model according to an exemplary embodiment. This embodiment is based on the above embodiments and further provides a specific implementation method for determining a memory detection model. Figure 2 As shown, the method includes:

[0101] In step S210, raw data statistics of account data stored in at least three memory storage systems are obtained, where the raw data of the account data is data directly stored and accessed in the memory according to business needs, and the raw data statistics of the account data is the storage byte length of the raw data.

[0102] In step S220, the actual space occupancy of the memory space in each of the memory storage systems is obtained, where the actual space occupancy is the size of the memory space actually occupied in the memory storage system.

[0103] As described in the above embodiment, the obtained raw data statistics and actual space occupancy can be aggregated according to the Redis cluster through the data warehouse tool (Hive) to obtain the total number of key-value pairs and the size of the raw data statistics for each Redis cluster. Finally, the training data set is obtained and can be output as a parquet file, which can specifically contain the following fields: cluster name, number of primary storage nodes, number of key-value pairs, size of raw data statistics, and actual space occupancy. Based on the files of the training data set, the memory detection model is fitted to solve the function parameters, such as a, b, and c in the above formula.

[0104] Optionally, before fitting the memory detection model, you can also analyze and clean up outliers in the raw data statistics and actual space usage. For example, some Redis clusters may store data that has no account, which can be cleared from the raw data statistics.

[0105] As a complete example of collecting raw data statistics and actual space usage, when collecting raw data statistics and actual space usage, Spark RedisScanner can be used to batch scan the data stored in the Redis cluster for each account (or each business); query the MySQL database metadata to obtain the Redis cluster and key prefix for each account. Specifically, by accessing the interface of the Redis management platform, the number and address of the primary storage node / backup storage node instances of each Redis cluster and the actual memory usage of each Redis cluster can be obtained; a backup storage node in the cluster can be selected for scanning, and the result can be multiplied by the number of all backup storage nodes to obtain capacity-related indicators for each account; Spark parallel tasks can be used to simultaneously scan the backup storage nodes of each Redis cluster and multiply the result by the number of backup storage nodes to obtain the number of keys and the size of raw data statistics for each account; the result can be output as a Hive table containing the following fields: Redis cluster name, number of master nodes, account ID, number of keys, size of raw data statistics, and actual memory usage. Specifically, Hive table data can be aggregated according to the Redis cluster to obtain the total number of keys and the size of the raw data statistics for each Redis cluster. Finally, the training dataset is obtained and output as a parquet file containing the following fields: cluster name, number of master nodes, number of keys, size of the raw data statistics, and actual memory usage.

[0106] In step S230, the original data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the least squares method is used to fit the parameters of the set memory detection model so that the residual function of the fitting result converges, wherein the memory detection model is used to detect the memory space occupancy based on the original data statistics of the account data stored in the memory storage system, and the set memory detection model is a linear function.

[0107] In this embodiment, a linear function is used as the memory detection model, which can be quickly fitted and obtain good detection accuracy with low computational complexity. During training and learning, the original data statistics and actual space occupancy of multiple memory storage systems are generally collected for training. The data volume of different memory storage systems may vary greatly, so when the least squares method is used for fitting, a residual function is used to calculate the error ratio between the detection value and the actual value as the residual. For example, the leastsq least squares fitting function (Levenberg-Marquardt algorithm) in the Python scipy.optimize package can be used to perform the fitting operation.

[0108] There are many ways to express the linear relationship between the original data statistics and the actual space occupancy. For example, if the memory storage system is a key-value pair storage system (such as a Redis cluster), the optional formula of the memory detection model is as follows:

[0109] Detection memory usage (estimatedSize) = a * number of key-value pairs (kvCount) + b * amount of data stored in the key-value pairs (rawSize) + c * number of storage nodes (masterNum);

[0110] Where a, b, and c are parameters of the memory detection model. kvCount represents the number of keys in the Redis cluster; rawSize represents the raw data size in the Redis cluster, including the byte length of the key and value; masterNum represents the number of master storage nodes in the Redis cluster, which is also equal to the number of backup storage nodes. The number of key-value pairs and the amount of data stored in these key-value pairs refer to the number of key-value pairs and the amount of data stored in these key-value pairs across all master nodes in the Redis cluster.

[0111] Optionally, the formula of the residual function is as follows:

[0112] Residual value = (detected memory usage - actual memory usage) / actual memory usage.

[0113] The residual value (residual) is the ratio of the difference between the detected value and the actual value to the actual value. The actual memory usage (actualSize) is the actual memory usage.

[0114] The technical solution of this embodiment determines the memory detection model by fitting a linear function using the least squares method, which can quickly and accurately determine the memory detection model at a low computational cost.

[0115] Based on this embodiment, preferably, after the step of determining all parameters in the memory detection model, the method further includes:

[0116] When it is determined that a storage memory model change event occurs, the steps of obtaining the original data statistics and the actual space occupancy and training to determine the parameters in the memory detection model are triggered to be re-executed.

[0117] In real-world environments, memory storage systems, their operating environments, and the business systems they support are subject to constant change and upgrades, necessitating corresponding changes to the memory storage system's spatial memory detection model. The technical solution of the disclosed embodiments allows for rapid retraining of the memory detection model with minimal computational effort, thereby maintaining the accuracy of the memory storage system's spatial memory detection.

[0118] The storage memory model change events mentioned above may include various situations. As long as they affect the possibility of additional memory overhead in the memory space, the memory detection model needs to be updated.

[0119] Specifically, the storage memory model change event may include at least one of the following:

[0120] First, the version of the memory storage system has changed; for example, the node sharding situation and node space balancing rules of the memory storage system have changed.

[0121] Second, the compilation parameters of the memory storage system change; for example, the compilation parameters of the memory storage system change.

[0122] Third, the operating parameters of the memory storage system change; for example, the operating parameters of the memory storage system are modified.

[0123] Fourth, the operating environment of the memory storage system changes; for example, the CPU version, operating system version, etc. of the memory storage system changes.

[0124] Fifth, changes in the distribution of data characteristics of account data stored in the memory storage system. Data characteristics include data type and / or the size of a single data entry. For example, account data may contain both numeric and non-numeric data. As the business grows, the proportion of numeric data may decrease, while the proportion of non-numeric data may increase. This may cause changes in the memory overhead incurred when storing this account data. Alternatively, account data can be divided into several length categories, such as short data and long data. When the ratio of long data to short data changes, memory overhead may also change. These factors may trigger updates to the memory detection model.

[0125] Optionally, when determining that a storage memory model change event occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training to determine parameters in the memory detection model includes at least one of the following steps:

[0126] When it is determined that a version change of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0127] When determining that a change in the compilation parameters of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0128] When it is determined that a change in the operating parameters of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0129] When it is determined that a change occurs in the operating environment of the memory storage system, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model;

[0130] When it is determined that a change in the data feature distribution of the account data stored in the memory storage system occurs, the steps of obtaining the original data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model are triggered, wherein the data features include the data type and / or the amount of a single data item.

[0131] Of course, the memory detection model can also be updated regularly to maintain its detection accuracy.

[0132] The embodiments of the present disclosure adopt the above technical solution, which has many advantages for memory storage systems supporting Internet services:

[0133] 1. For new businesses that will use the Redis database, this memory detection model can be used to estimate the memory usage after importing the account data into the Redis database based on the raw data statistics, enabling accurate memory resource planning. For example, for new business accounts that want to apply for a Redis cluster, the memory detection model can be used to detect the memory capacity occupied by the raw data statistics after importing them into the Redis cluster, allowing the user to select a Redis cluster with appropriate remaining capacity.

[0134] 2. For existing businesses using the Redis database, this memory detection model can be used to continuously track the growth of business data volume. On this basis, quota restrictions and billing can be implemented to encourage rational resource usage. For example, Spark RedisScanner can be used to batch scan the raw data statistics of each Redis cluster every morning. The actual memory usage can be detected based on the memory detection model, and a memory usage bill email can be sent to the business account to remind it of its resource usage.

[0135] 3. Compared with the estimation scheme determined by analyzing the memory model, the estimation accuracy is greatly improved when the memory model changes, and the error can usually be reduced from 25% to 60% to about 5%.

[0136] 4. This solution only requires statistical information such as the number of key-value pairs and data size of the original data to complete the memory usage assessment. Compared with the memory model solution, it is simpler to use.

[0137] 5. In actual applications, different memory storage clusters can support different businesses. Each business can be regarded as a memory demander. For ease of management, the data of a memory demander can be supported in one memory storage cluster. There may also be situations where multiple businesses share the same Redis cluster. Generally, each business will have a different key prefix. Based on the technical solution of the embodiment of the present disclosure, the original data statistics can be directly determined based on the key prefix of the business, and then the memory usage can be determined based on the memory detection model. In this way, the memory usage of different businesses can be determined.

[0138] Figure 3 FIG. 1 is a block diagram of a training device for a memory detection model according to an exemplary embodiment. Figure 3 The device includes a statistical value acquisition unit 310, an actual space occupancy acquisition unit 320 and a model training unit 330.

[0139] Among them, the statistical value acquisition unit 310 is configured to obtain the original data statistical values ​​of the account data stored in at least three memory storage systems, where the original data of the account data is the data directly stored and accessed in the memory according to business needs, and the original data statistical value of the account data is the storage byte length of the original data; the actual space occupancy acquisition unit 320 is configured to obtain the actual space occupancy of the memory space occupied by each of the memory storage systems, where the actual space occupancy is the size of the memory space actually occupied in the memory storage system; the model training unit 330 is configured to use the original data statistical values ​​and actual space occupancy corresponding to each of the memory storage systems as training samples, and train and learn the parameters of the set memory detection model to determine all parameters in the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the original data statistical values ​​of the account data stored in the memory storage system.

[0140] The disclosed embodiment can determine the memory detection model by fitting and learning the memory detection model using the raw data statistics and actual space occupancy in the memory storage system. This method is not determined by analyzing the memory model of the memory storage system. There is no strict dependency on the version of the memory storage system itself, the operating environment, and the account data, and it has universal applicability. When it is necessary to understand the space occupancy of the memory storage system, the actual space occupancy can be quickly and accurately detected based on the raw data statistics of the account data, which has guiding significance for the maintenance and management of the memory space.

[0141] In an optional implementation of the embodiment of the present disclosure, if the memory detection model is a linear function, the model training unit 330 is configured to:

[0142] The original data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the least squares method is used to fit the parameters of the set memory detection model so that the residual function of the fitting result converges.

[0143] In an optional implementation of the embodiment of the present disclosure, if the memory storage system is a key-value pair storage system, the formula of the memory detection model is as follows:

[0144] Detection memory usage = a * number of key-value pairs + b * amount of data stored in the key-value pairs + c * number of storage nodes;

[0145] Among them, a, b, and c are the parameters of the memory detection model.

[0146] In an optional implementation of the embodiment of the present disclosure, the statistical value acquisition unit 310 includes:

[0147] The traversal access subunit is configured to, for each memory storage system, perform a traversal access on any storage node of a memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the storage node; each memory storage cluster includes at least two storage nodes, and the memory storage cluster adopts a node space balancing occupancy rule;

[0148] The first data determination subunit is configured to determine the raw data statistic value of the memory storage cluster according to the raw data statistic value of a storage node and the number of storage nodes in the memory storage cluster.

[0149] In an optional implementation of the embodiment of the present disclosure, the statistical value acquisition unit 310 includes:

[0150] A backup node scanning subunit is configured to, for each memory storage system, perform traversal access to backup storage nodes in a memory storage cluster serving as the memory storage system to determine raw data statistics of account data in the backup storage nodes; wherein the memory storage cluster includes at least one node shard, each node shard includes a primary storage node and at least one backup storage node, the primary storage node is used to provide online services, and the backup storage node is used to synchronize data of the primary storage node for backup;

[0151] The second data determination subunit is configured to determine the original data statistical value of the memory storage cluster according to the original data statistical value of the backup storage node.

[0152] In an optional implementation of the embodiment of the present disclosure, the training device of the memory detection model further includes:

[0153] The meta information scanning unit is configured to obtain the cluster name and the number of storage nodes of each of the memory storage clusters by scanning the database meta information of each of the memory storage systems.

[0154] In an optional implementation of the embodiment of the present disclosure, the training device of the memory detection model further includes:

[0155] The function update unit is configured to trigger the steps of re-executing the steps of obtaining the original data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model when determining that a storage memory model change event occurs.

[0156] In an optional implementation of the embodiment of the present disclosure, the function updating unit includes:

[0157] A first function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining parameters in the memory detection model when determining that a version change of the memory storage system occurs;

[0158] A second function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters of the memory detection model when determining that a compilation parameter change of the memory storage system occurs;

[0159] A third function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters of the memory detection model when determining that a change in the operating parameters of the memory storage system occurs;

[0160] a fourth function updating subunit configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining parameters in the memory detection model when determining that a change in the operating environment of the memory storage system occurs;

[0161] The fifth function update sub-unit is configured to trigger the steps of re-executing the steps of obtaining the original data statistics and the actual space occupancy, and training to determine the parameters in the memory detection model when it is determined that the data feature distribution of the account data stored in the memory storage system has changed, wherein the data features include the data type and / or the amount of a single data item.

[0162] The technical solution of the disclosed embodiment has many advantages: it uses machine learning methods to replace the traditional memory model capacity calculation method, solves the errors in estimating memory capacity occupancy caused by different versions, environments, and cluster deployment methods, and enables the memory occupancy estimation strategy to meet the requirements in various situations and maintain a small estimation error.

[0163] The disclosed embodiments offer multiple solutions for fitting linear function equations, including the Levenberg-Marquardt least squares method. This method uses a more reasonable error ratio as the residual, resulting in better evaluation results for clusters with significantly different storage capacities. Compared to directly using the difference in estimated capacity as the residual, the evaluation error for large-capacity Redis clusters is smaller, while the evaluation error for small-capacity Redis clusters is larger.

[0164] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0165] Figure 4FIG. 1 is a structural diagram of an electronic device according to an exemplary embodiment. Figure 4 As shown, the electronic device includes:

[0166] one or more processors 410,

[0167] Figure 4 A processor 410 is taken as an example;

[0168] Memory 420;

[0169] The processor 410 and the memory 420 in the device can be connected via a bus or other means. Figure 4 The bus connection is taken as an example.

[0170] The memory 420 is a non-transitory computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as program instructions / modules corresponding to a training method for a memory detection model in an embodiment of the present disclosure (for example, Figure 3 The processor 410 executes the software programs, instructions, and modules stored in the memory 420 to execute various functional applications and data processing of the computer device, thereby implementing a training method for a memory detection model of the above-mentioned method embodiment, namely:

[0171] Obtaining raw data statistics of account data stored in at least three memory storage systems, where the raw data of the account data is data directly stored and accessed in memory according to business needs of the account, and the raw data statistics of the account data is the storage byte length of the raw data;

[0172] Obtaining the actual space occupancy of the memory space in each of the memory storage systems, where the actual space occupancy is the size of the memory space actually occupied in the memory storage system;

[0173] The raw data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the parameters of the set memory detection model are trained and learned to determine all parameters in the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the raw data statistics of the account data stored in the memory storage system.

[0174] The memory 420 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 420 may optionally include a memory remotely located relative to the processor 410, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 420 including instructions. The instructions can be executed by the processor 410 of the electronic device to perform the above method. Alternatively, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0176] In an exemplary embodiment, a computer program product is also provided for use in conjunction with an electronic device. The computer program product includes a computer-readable storage medium and a computer program mechanism embedded therein. After the program is loaded and executed by a computer, the training method of the memory detection model as described in any embodiment of the present disclosure can be implemented.

[0177] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0178] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a memory detection model based on a Redis cluster, characterized in that: include: Obtaining raw data statistics of account data stored in at least three memory storage systems, where the raw data of the account data is data directly stored and accessed in memory according to business needs of the account, and the raw data statistics of the account data is the storage byte length of the raw data; Obtaining the actual space occupancy of the memory space in each of the memory storage systems, where the actual space occupancy is the size of the memory space actually occupied in the memory storage system; Using the raw data statistics and actual space occupancy corresponding to each of the memory storage systems as training samples, training and learning parameters of a set memory detection model to determine all parameters of the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the raw data statistics of the account data stored in the memory storage system; Wherein, the memory storage system is a Redis memory storage cluster.

2. The method according to claim 1, characterized in that If the memory detection model is a linear function, the raw data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples to train and learn the parameters of the set memory detection model, including: The original data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the least squares method is used to fit the parameters of the set memory detection model so that the residual function of the fitting result converges.

3. The method according to claim 2, characterized in that If the memory storage system is a key-value pair storage system, the formula of the memory detection model is as follows: Detection memory usage = a * number of key-value pairs + b * amount of data stored in the key-value pairs + c * number of storage nodes; Among them, a, b, and c are the parameters of the memory detection model.

4. The method according to claim 1, wherein The obtaining of raw data statistics of account data stored in at least three memory storage systems includes: For each memory storage system, traverse and access any storage node of a memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the storage node; each memory storage cluster includes at least two storage nodes, and the memory storage cluster adopts a node space balancing rule; The raw data statistic of the memory storage cluster is determined according to the raw data statistic of a storage node and the number of storage nodes in the memory storage cluster.

5. The method according to claim 1, characterized in that The obtaining of raw data statistics of account data stored in at least three memory storage systems includes: For each memory storage system, a traversal access is performed on the backup storage nodes in the memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the backup storage nodes; wherein the memory storage cluster includes at least one node shard, each node shard includes a primary storage node and at least one backup storage node, the primary storage node is used to provide online services, and the backup storage node is used to synchronize data of the primary storage node for backup; The original data statistics of the memory storage cluster are determined according to the original data statistics of the backup storage node.

6. The method according to claim 4 or 5, characterized in that Before obtaining the raw data statistics of the account data stored in the at least three memory storage systems, the method further includes: The cluster name and the number of storage nodes of each memory storage cluster are obtained by scanning the database meta information of each memory storage system.

7. The method according to claim 1, characterized in that After the step of determining all parameters in the memory detection model, the method further includes: When it is determined that a storage memory model change event occurs, the steps of obtaining the original data statistics and the actual space occupancy and training to determine the parameters in the memory detection model are triggered to be re-executed.

8. The method according to claim 7, characterized in that The triggering of re-execution of the steps of obtaining the raw data statistics and the actual space occupancy and training to determine the parameters in the memory detection model when determining that a storage memory model change event occurs includes at least one of the following steps: When it is determined that a version change of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model; When determining that a change in the compilation parameters of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model; When it is determined that a change in the operating parameters of the memory storage system occurs, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model; When it is determined that a change occurs in the operating environment of the memory storage system, triggering re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model; When it is determined that a change in the data feature distribution of the account data stored in the memory storage system occurs, the steps of obtaining the original data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model are triggered, wherein the data features include the data type and / or the amount of a single data item.

9. A training device for a memory detection model based on a Redis cluster, characterized in that: include: a statistical value acquisition unit configured to acquire raw data statistical values ​​of account data stored in at least three memory storage systems, wherein the raw data of the account data is data directly stored and accessed in the memory according to business needs of the account, and the raw data statistical value of the account data is the storage byte length of the raw data; an actual space occupancy acquisition unit configured to acquire an actual space occupancy of the memory space in each of the memory storage systems, wherein the actual space occupancy is the size of the memory space actually occupied in the memory storage system; a model training unit configured to train and learn parameters of a set memory detection model using the raw data statistics and actual space occupancy corresponding to each of the memory storage systems as training samples, so as to determine all parameters of the memory detection model, wherein the memory detection model is used to detect the memory space occupancy based on the raw data statistics of the account data stored in the memory storage system; Wherein, the memory storage system is a Redis memory storage cluster.

10. The device according to claim 9, characterized in that If the memory detection model is a linear function, the model training unit is configured to: The original data statistics and actual space occupancy corresponding to each of the memory storage systems are used as training samples, and the least squares method is used to fit the parameters of the set memory detection model so that the residual function of the fitting result converges.

11. The device according to claim 10, characterized in that If the memory storage system is a key-value pair storage system, the formula of the memory detection model is as follows: Detection memory usage = a * number of key-value pairs + b * amount of data stored in the key-value pairs + c * number of storage nodes; Among them, a, b, and c are the parameters of the memory detection model.

12. The device according to claim 9, characterized in that The statistical value acquisition unit includes: The traversal access subunit is configured to, for each memory storage system, perform a traversal access on any storage node of a memory storage cluster serving as the memory storage system to determine the raw data statistics of the account data in the storage node; each memory storage cluster includes at least two storage nodes, and the memory storage cluster adopts a node space balancing occupancy rule; The first data determination subunit is configured to determine the raw data statistic value of the memory storage cluster according to the raw data statistic value of a storage node and the number of storage nodes in the memory storage cluster.

13. The device according to claim 9, characterized in that The statistical value acquisition unit includes: A backup node scanning subunit is configured to, for each memory storage system, perform traversal access to backup storage nodes in a memory storage cluster serving as the memory storage system to determine raw data statistics of account data in the backup storage nodes; wherein the memory storage cluster includes at least one node shard, each node shard includes a primary storage node and at least one backup storage node, the primary storage node is used to provide online services, and the backup storage node is used to synchronize data of the primary storage node for backup; The second data determination subunit is configured to determine the original data statistical value of the memory storage cluster according to the original data statistical value of the backup storage node.

14. The device according to claim 12 or 13, characterized in that Also includes: The meta information scanning unit is configured to obtain the cluster name and the number of storage nodes of each of the memory storage clusters by scanning the database meta information of each of the memory storage systems.

15. The device according to claim 9, characterized in that Also includes: The function update unit is configured to trigger the steps of re-executing the steps of obtaining the original data statistics and the actual space occupancy, and training and determining the parameters in the memory detection model when determining that a storage memory model change event occurs.

16. The device according to claim 15, characterized in that The function updating unit includes: A first function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining parameters in the memory detection model when determining that a version change of the memory storage system occurs; A second function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters of the memory detection model when determining that a compilation parameter change of the memory storage system occurs; A third function updating subunit is configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining the parameters of the memory detection model when determining that a change in the operating parameters of the memory storage system occurs; a fourth function updating subunit configured to trigger re-execution of the steps of obtaining the raw data statistics and the actual space occupancy, and training and determining parameters in the memory detection model when determining that a change in the operating environment of the memory storage system occurs; The fifth function update sub-unit is configured to trigger the steps of re-executing the steps of obtaining the original data statistics and the actual space occupancy, and training to determine the parameters in the memory detection model when it is determined that the data feature distribution of the account data stored in the memory storage system has changed, wherein the data features include the data type and / or the amount of a single data item.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the memory detection model as described in any one of claims 1 to 7.

18. A storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the training method of the memory detection model as described in any one of claims 1 to 7.

Images