Automatic analysis and management method, system and device for data storage

Abstract

The invention relates to the technical field of database management, and discloses an automatic analysis and management method, system and device for data storage, and the key point of the technical scheme is that the method comprises the following steps: S1, constructing a migration strategy based on data real-time attributes; s2, in response to the analysis management instruction, evaluating the main service table to obtain an evaluation result; s3, matching the evaluation results according to a migration strategy, and determining target migration data; s4, executing a uniform distribution algorithm on the target migration data to obtain a sub-table serial number to be migrated; and S5, migrating the target migration data into the target sub-table according to the sub-table serial number, realizing an automatic analysis mechanism through a migration rule, intelligently judging a migration data range according to a multi-dimensional factor, and automatically executing migration operation, so that historical data which does not influence a main business process can be accurately screened out for migration, and the migration efficiency is improved. And the main service table always keeps high-efficiency operation, so that the overall performance of the system is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of database management, and more specifically, it relates to a method, system and device for automated analysis and management of data storage. Background Art

[0002] At present, with the rapid development of digital business, the database, as the core of data storage and management, is facing the severe challenge of exponential growth in the amount of data. As the data within the table continues to accumulate, the performance of conventional operations such as database insertion, deletion, modification, and query gradually declines, which has a serious negative impact on the response speed and stability of the system. For example, in some large e-commerce platforms, as the business data volume increased by 500% within one year, the query response time of the database extended from the original average of 0.5 seconds to more than 3 seconds, and the throughput of the system also decreased by about 40%, seriously affecting the user experience and the normal operation of the business.

[0003] Currently, database sharding is a commonly used means to address the problem of data volume growth. By dispersing data storage, it effectively reduces the amount of data in a single table and improves the overall performance to a certain extent. However, in actual business scenarios, database sharding exposes many drawbacks.

[0004] Taking MySQL database sharding as an example, in practice, middleware (such as MyCAT and Sharding-JDBC) is often used to achieve it. MyCAT routes data based on configuration rules, such as primary keys and modulo algorithms, and disperses the data storage in different databases and tables. Sharding-JDBC is integrated at the application layer, and developers can customize the database sharding strategy. These existing solutions have many problems in actual application. Taking the credit management system of a financial institution as an example, when displaying the background management list, after adopting MySQL database sharding, it is necessary to integrate data from multiple sharded databases and tables, which greatly increases the complexity of queries and resource consumption. Before optimizing with database sharding, it only took 0.2 seconds to query the credit customer list of this institution. However, after database sharding, since it involves extracting and integrating data from 5 different databases and 10 sharded tables, the query time extended to 1.5 seconds, and the query efficiency decreased by more than 8 times. Moreover, the server CPU resources occupied by the query operation soared from 5% to 30%, seriously affecting the overall operation efficiency of the system.

[0005] These problems indicate that the existing database sharding technology cannot fundamentally solve the problem of database performance decline when dealing with data volume growth, and also brings new complex problems. To break this dilemma, a new idea and method are needed. Summary of the Invention

[0006] The objective of the present invention is to provide a method, system and device for automated analysis and management of data storage, which realizes an automated analysis mechanism through migration rules, intelligently determines the range of migrated data based on multi-dimensional factors and automatically executes the migration operation, can accurately screen out historical data that does not affect the main business process for migration, ensure that the main business table always runs efficiently, and thus effectively improve the overall performance of the system.

[0007] The above technical objective of the present invention is achieved through the following technical solutions: A method for automated analysis and management of data storage, including the following steps:

[0008] S1. Construct a migration strategy based on the real-time attributes of data;

[0009] S2. Respond to the analysis and management instruction, evaluate the main business table, and obtain an evaluation result;

[0010] S3. Match the evaluation result according to the migration strategy to determine the target migration data;

[0011] S4. Execute a uniform distribution algorithm on the target migration data to obtain the sub-table numbers to be migrated;

[0012] S5. According to the sub-table numbers, migrate the target migration data to the target sub-tables.

[0013] As a preferred technical solution of the present invention, the real-time attributes of data include: creation time, update time, business status, business attributes, data volume;

[0014] The evaluation result includes the creation time, update time, business status, business attributes, and data volume of each item of data in the main business table;

[0015] The migration rules included in the migration strategy include: regular cleaning rules, time window screening rules, business completion determination rules, status conversion trigger rules, main table threshold control rules, and sub-table balance rules.

[0016] As a preferred technical solution of the present invention, in S1, after constructing the migration strategy, a decision tree model with each migration rule in the migration strategy as a pivot is constructed;

[0017] The decision tree model is used to input the evaluation result, output the decision result, and determine the target migration data.

[0018] As a preferred technical solution of the present invention, in S3, after determining the target migration data, through the association rule mining algorithm, analyze the business association data of each item of target migration data, and judge whether the business association data all conforms to the migration strategy. If so, execute S4 on the target migration data. If not, remove this item of data from the target migration data.

[0019] As a preferred technical solution of the present invention, in S4, the uniform distribution algorithm includes modulo processing of hash values based on dynamic weight adjustment, and the process is as follows:

[0020] Obtain the unique number contained in each piece of target migration data, determine the number of sub-tables corresponding to the main business table, obtain the current dynamic weight factor information, and find the dynamic weight coefficient matching the target migration data;

[0021] Convert the format of the unique number of the target migration data to obtain a numerical number in a unified format;

[0022] Perform an operation on the dynamic weight coefficient and the data number to obtain a final numerical number;

[0023] Perform modulo operation on the final numerical number based on the number of sub-tables to obtain the sub-table serial number;

[0024] Determine the target sub-table to be migrated according to the sub-table serial number.

[0025] As a preferred technical solution of the present invention, in S3, after determining the target migration data, sort each piece of target migration data according to the priority rule to obtain a data list;

[0026] In S4, modulo processing of hash values is sequentially performed on the target migration data according to the order of the data list.

[0027] As a preferred technical solution of the present invention, in S5, read the target migration data from the main business table and perform integrity verification. If the data is complete, temporarily store the target migration data in the temporary storage area;

[0028] Create a record space in the target sub-table corresponding to the sub-table serial number;

[0029] Write the target data in the temporary storage area into the record space in the target sub-table, and during the writing process, perform consistency verification on the written target migration data and the original data in the main business table.

[0030] As a preferred technical solution of the present invention, during the execution of S1 - S5, continuously record the management operation logs, monitor the status of the main business table and its sub-tables, and set the dynamic weight coefficients of the data in the current dynamic weight factor information according to the status of the main business table and its sub-tables and the real-time attributes of the data;

[0031] The management operation logs include migration time, data volume, the location of the source data in the main business table, and the target sub-table information after migration.

[0032] A data storage automated analysis and management system, comprising:

[0033] A rule configuration module for constructing a migration strategy based on the real-time attributes of data;

[0034] A data evaluation module that, in response to an analysis management instruction, evaluates the main business table to obtain an evaluation result;

[0035] A rule matching module that matches the evaluation result according to the migration strategy to determine the target migration data;

[0036] A data allocation module that executes a uniform allocation algorithm on the target migration data to obtain the sub-table numbers to be migrated;

[0037] A data migration module that migrates the target migration data to the target sub-table according to the sub-table numbers;

[0038] An operation monitoring module for continuously recording management operation logs, monitoring the status of the main business table and its sub-tables, and setting the dynamic weight coefficients of the data in the current dynamic weight factor information according to the status of the main business table and its sub-tables and the real-time attributes of the data.

[0039] A data storage automatic analysis and management device, including: a processor and a memory, the memory stores a computer program executable by the processor, and the processor implements the above method when executing the computer program.

[0040] In summary, the present invention has the following beneficial effects: By focusing on the main business table and only migrating historical data, it avoids excessive splitting of business data like database sharding and sub-tabling, reducing the complexity of data management at the source. At the same time, through the migration rules, an automatic analysis mechanism is realized. Based on multi-dimensional factors, the range of migration data is intelligently determined and the migration operation is automatically executed. It can accurately screen out historical data that does not affect the main business process for migration, ensuring that the main business table always runs efficiently, thereby effectively improving the overall performance of the system and providing an innovative and effective solution to solve the problem of database performance degradation.

[0041] In the present invention, the main process business still uses single-table operations, and at the same time, a set of several historical backup tables is created for the main business table. The data that meets specific conditions, that is, the migration rules, is migrated from the main business table to one of the sub-tables. The overall solution is based on the concept of historical data sub-tabling. On the premise of ensuring the singularity of the main business process table, it reduces the data volume of the main business table, thereby improving system performance, avoiding complex cross-database and cross-table operations, reducing system resource consumption, improving query performance, ensuring stable and efficient operation of the business, enhancing user experience, and contributing to the healthy development of enterprise business. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is the method flow chart of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below with reference to the accompanying drawings.

[0044] As Figure 1 shown, the present invention provides a method for automated analysis and management of data storage, including the following steps:

[0045] S1. Construct a migration strategy based on the real-time attributes of data;

[0046] The real-time attributes of data include: creation time, update time, business status, business attributes, and data volume;

[0047] Specifically, the migration rules included in the migration strategy include: regular cleaning rules, time window screening rules, business completion determination rules, status conversion trigger rules, main table threshold control rules, and sub-table balance rules.

[0048] After constructing the migration strategy, construct a decision tree model with each migration rule in the migration strategy as the fulcrum;

[0049] The decision tree model is used to input the evaluation results, output the decision results, and determine the target migration data. The accuracy and reliability of the decision tree model largely depend on the quality and richness of the training data. In the present invention, the training data source of the decision tree is extensive and comprehensive, covering information in multiple dimensions.

[0050] From the time dimension, the creation time and the last update time of the data are important features. In the credit business scenario, the creation time of the credit application is a key indicator. If the creation time of a credit application exceeds 1 year and there is no update record in the past 6 months, then this type of data is very likely to meet the migration conditions. Because data that has not been updated for a long time has lower real-time and importance in the current business operation, it can be considered to migrate it to the historical backup table to reduce the burden on the main business table.

[0051] In terms of business status, key status information in various business processes is collected. In the credit system, the credit status is a core feature, including different statuses such as "approved", "rejected", and "archived". The credit application data in the "rejected" and "archived" statuses usually indicates that the business process has ended and has a lower requirement for real-time. These data are more likely to meet the migration conditions and can provide important decision-making basis for the decision tree model. After comprehensive evaluation, the data that may be determined to be suitable for migration.

[0052] The data volume also plays a crucial role in the training of the decision tree model. Not only the data volume of the main business table is concerned, but also the data volume of relevant sub-tables will be deeply analyzed. Taking the credit business as an example, the growth trend of the data volume in the credit main table is an important indicator. If the data volume in the credit main table grows rapidly within a short period of time and reaches a certain threshold, such as 100,000 records, and in the associated tables such as quota, contract, and loan related to it, the associated data all meet certain conditions, then these data are very likely to meet the migration conditions. Because the activity of these data in the sub-tables is low and their real-time support for the current business is limited, migrating them will not have a great impact on the core business process, but will help optimize the storage and query performance of the system.

[0053] Business attributes are also an important part of the training data, containing rich information such as product names and customer types. In the field of financial credit business, different financial products correspond to different credit-granting rules and data processing methods. For example, there are significant differences in the approval process, risk assessment, and data management between personal consumer loans and corporate business loans. The attribute of product name can help the decision tree model better identify the data characteristics of different financial products, so as to more accurately judge whether the data meets the migration conditions. The data generated by new customers at the initial stage of registration may have high research value and should not be migrated prematurely; while the data generated by some long-inactive accounts among old customers may be more suitable for migration processing after comprehensive evaluation. These business attribute information can help the decision tree model comprehensively analyze the data from multiple angles and improve the accuracy of judging the data migration conditions.

[0054] After collecting the above rich raw data and various labels, training the decision tree model can ensure that the data output by the decision tree model is of reliable quality and accurate. The decision tree algorithm has the characteristics of being intuitive and easy to understand, and can clearly show the process and basis of data migration decisions, facilitating the adjustment and optimization of the migration data scope.

[0055] Regarding the various migration rules in the migration strategy,

[0056] Regular cleaning rule: Set the cleaning cycle according to business characteristics and data activity, such as monthly, quarterly or annually. At the end of each cycle, scan the main business table and migrate the data earlier than a specific time point. For example, at the beginning of each month, migrate the data created before the end of last month and with no update operations in the past month to the historical sub-table.

[0057] Time window screening rule: Set different time windows for different types of data. For example, for transaction data, if there have been no transaction-related operations, including payments, refunds, shipments, etc., for more than 90 days, then migrate it; for log data, records over 30 days can be migrated.

[0058] Business completion determination rule: When the business process reaches a clear completion state, data migration is triggered. In the credit system, when the credit granting status is "rejected", "archived", etc., the application data is migrated.

[0059] Status conversion trigger: By monitoring the conversion of the business status, when the status changes from "processing" to specific statuses such as "archived", "terminated", etc., the corresponding data is migrated.

[0060] Master table threshold control rule: Set the upper limit threshold for the data volume of the main business table. When the data volume of the main business table reaches or exceeds this threshold, the data migration process is started. The selection of migrated data gives priority to data that meets the time or business status rules. If the data volume target is still not reached, then according to a certain sorting rule, such as the order of data creation time, the migration is carried out.

[0061] Sub-table balancing strategy: Monitor the data volumes of each historical sub-table to ensure that the data is evenly distributed among 32 sub-tables. Hash value modulo operation is performed in this link to ensure data evenness.

[0062] More specifically, the hash value modulo operation is also optimized: Before the hash operation, the business number is preprocessed. For example, for a business number containing letters and numbers, the letters are first converted to numbers according to specific rules, such as A corresponding to 1, B corresponding to 2, etc., and then combined with the original number part. Then a dynamic weight coefficient is introduced to adjust the weight according to the business type or data generation time, making the hash value more random. Finally, take the modulo of 32 to ensure that the data is evenly distributed among the sub-tables.

[0063] S2. In response to the analysis management instruction, evaluate the main business table to obtain an evaluation result; the evaluation result includes the creation time, update time, business status, business attributes, and data volume of each item of data in the main business table; it is used to comprehensively evaluate the data in the main business table, and can also be used to analyze historical data and current business trends, predict future data growth, and provide a basis for formulating the migration strategy.

[0064] It should be noted that the main business table of the present invention has several sub-tables for migrating and storing various historical data.

[0065] S3. Match the evaluation result according to the migration strategy to determine the target migration data;

[0066] In S3, after determining the target migration data, through the association rule mining algorithm, analyze the business associated data of each item of target migration data, and judge whether the business associated data all conforms to the migration strategy. If so, execute S4 on the target migration data; if not, remove this item of data from the target migration data.

[0067] Specifically, the association rule mining algorithm can help discover potential patterns from a large amount of data, provide a basis for the scope of migrated data, analyze the association relationships between different data attributes in the database, and find frequently occurring data item sets. For example, the credit records to be migrated are marked through a decision tree model. However, in the entire business process, it includes: credit-granting - quota - contract - loan disbursement - repayment - imaging documents, etc. It is necessary to lock the data associated with other business scenarios in the data stream according to the credit records to check whether the migration conditions are met. For example, although the credit filing meets the migration requirements, if the withdrawal corresponding to this credit is overdue due to the customer's failure to repay the loan in a timely manner, then all the associated data on this link will not meet the migration rules.

[0068] In S3, after determining the target migration data, according to the priority rules, each piece of target migration data is sorted to obtain a data list; the priority rules are used to formulate the priorities of various data based on time priority, the urgency of business status, etc.

[0069] In S4, for the target migration data, the uniform distribution algorithm is executed to obtain the sub-table numbers to be migrated;

[0070] In S4, according to the order of the data list, the modulo operation of the hash value is performed on the target migration data in sequence.

[0071] In S4, the uniform distribution algorithm includes the modulo operation of the hash value based on dynamic weight adjustment, and its process is as follows:

[0072] In S41, obtain the unique number included in each piece of target migration data, such as the customer number corresponding to the data, determine the number of sub-tables corresponding to the main business table, obtain the current dynamic weight factor information, and find the dynamic weight coefficient matching the target migration data;

[0073] In S42, perform format conversion on the unique number of the target migration data to obtain a numerical number in a unified format; this step is a preprocessing of the unique number of the target migration data. For example, for a business number containing letters and numbers, the letters will be strictly converted to numbers according to the preset mapping rules. Taking English letters as an example, from A to Z correspond to the numbers 1 to 26 respectively, and other special characters also have corresponding conversion rules. After completing the letter conversion, it is merged with the original number part to unify the number format and provide a more standardized data input for the subsequent hash operation.

[0074] In S43, perform an operation on the dynamic weight coefficient and the data number to obtain the final numerical number; for example,

[0075] In S44, perform a hash modulo operation on the final numerical number based on the number of sub-tables to obtain the sub-table number;

[0076] S45. Determine the target sub-table to be migrated according to the sub-table sequence number.

[0077] Regarding the dynamic weight coefficient, it is suitable for further improving the randomness of the hash value. This dynamic weight coefficient will be adjusted according to other parameters such as business type and data generation time.

[0078] For example, in the credit field, during annual promotional activities such as the "New Year's Day Good Start", the amount of newly generated loan data is huge and the system's real-time processing requirements are high. At this stage, the present invention will automatically increase the weight of these loan data. Specifically, the extent of weight adjustment is determined by analyzing historical promotional activity data. Assuming that in normal times, the weight coefficient of loan data is 1, during promotional activities, this dynamic weight coefficient will be increased to between 1.5 and 2.5, and the specific value will be dynamically adjusted according to the scale of the activity and the fluctuation of historical data. At the same time, the time of data generation will also affect the weight. The closer the generation time, the higher the weight. Time decay calculation is performed in minutes. Every 10 minutes, the weight decays by 0.05, and the decay coefficient is adjusted according to the actual business situation. This can ensure that the recently generated data is more random during the hash operation, avoiding excessive concentration of data in some sub-tables, resulting in data imbalance between all sub-tables.

[0079] S5. Migrate the target migration data to the target shard table according to the shard table sequence number, including:

[0080] S51. Read the target migration data from the main business table and perform integrity check. If the data is complete, temporarily store the target migration data in a temporary storage area to check whether the data is missing or damaged.

[0081] S52. Create a record space in the target sub-table corresponding to the sub-table sequence number;

[0082] S53. Write the target data in the temporary storage area to the record space in the target sub-table. During the writing process, perform a consistency check on the written target migration data and the original data in the main business table to ensure that the written data is consistent with the original data in the main business table.

[0083] In S51, if the integrity check fails, the subsequent migration operation cannot be performed, and the following processing is performed:

[0084] First, the incomplete data is recorded in a log for subsequent analysis and processing. These logs include the incomplete data identifier, error type, and timestamp.

[0085] For important or large amounts of incomplete data, relevant data managers or system operation and maintenance personnel can be notified so that they can understand the situation in a timely manner and take appropriate measures;

[0086] If a relatively high proportion of incomplete data is found, it may be necessary to optimize the data processing flow, for example, improving the ways of data collection, transmission, and storage to enhance data integrity;

[0087] Data repair or deletion: For incomplete data that cannot be repaired, it can be considered to delete it from the system or transfer it to a dedicated "problem data" table for subsequent analysis and processing;

[0088] Update the migration strategy: According to the situation of incomplete data, it may be necessary to adjust the migration strategy, such as modifying the data screening conditions, adjusting the data volume threshold, etc., to ensure that the migrated data has high integrity.

[0089] Monitoring and alarming: Establish a real-time monitoring mechanism to continuously monitor data integrity. When a large amount of incomplete data is found, trigger the alarm mechanism to take timely measures.

[0090] Through the above steps, the situation of failed verification in the process of automated analysis and management of data storage can be effectively handled, ensuring the stable operation of the system and the accuracy of data.

[0091] Similarly, when performing consistency verification in S53, if the verification fails, the migration operation is already in progress at this time. Therefore, log the inconsistent data. If important data or a large amount of inconsistent data is involved, notify the relevant staff so that the staff can discover and handle it in a timely manner. At the same time, the optimization of the data processing flow, the update of the migration strategy, and the triggering of the monitoring and alarm mechanism are all processed according to the actual inconsistent data to ensure that abnormal situations can be discovered and handled in a timely manner, guaranteeing the stability of the system operation and data accuracy.

[0092] During the execution of S1 - S5, continuously record the management operation logs; the management operation logs include the migration time, data volume, the location of the source data in the main business table, the information of the target sub - table after migration, and other operation information. These log information is convenient for subsequent auditing and tracing. When data problems occur, the link and cause of the problem can be quickly located through the logs.

[0093] It will also monitor the status of the main business table and its sub - tables, and set the dynamic weight coefficients of the data in the current dynamic weight factor information according to the status of the main business table and its sub - tables and the real - time attributes of the data;

[0094] That is, establish a real - time monitoring mechanism. The monitoring frequency can be adjusted according to system performance and data volume, regularly check the data volume of each sub - table, or trigger it manually according to management requirements.

[0095] For example, if it is found that the data volume difference between any two sub-tables exceeds the preset adjustment threshold, the system will immediately start the weight factor recalibration program. During the recalibration process, factors such as the current business type, data generation time distribution, and recent data volume fluctuations will be comprehensively analyzed to re-determine the dynamic weight coefficient. In this way, the impact of data skew on system performance is effectively avoided, ensuring the efficient and stable operation of the system.

[0096] For example, if the business volume soars within a certain period, resulting in a sharp increase in the data of the main business table, then the time window screening will be adjusted. For example: the data that originally exceeded 1 year will be changed to the data that exceeds 6 months for migration.

[0097] The advantages of the technical solution of the present invention are as follows: By focusing on the main business table and only migrating historical data, it avoids excessive splitting of business data like database sharding and table partitioning, reducing the complexity of data management at the source. At the same time, through the migration rules, an automated analysis mechanism is realized. Based on multi-dimensional factors, the migration data range is intelligently determined and the migration operation is automatically executed. It can accurately screen out the historical data that does not affect the main business process for migration, ensuring that the main business table always runs efficiently, thereby effectively improving the overall performance of the system and providing an innovative and effective solution to solve the problem of database performance degradation.

[0098] In the present invention, the main process business still uses single-table operation, and at the same time, a set of historical backup tables is created for the main business table. The data that meets specific conditions, that is, the migration rules, is migrated from the main business table to one of the sub-tables. The overall solution is based on the concept of historical data sub-tables. On the premise of ensuring the singularity of the main business process table, the data volume of the main business table is reduced, thereby improving system performance, avoiding complex cross-database and cross-table operations, reducing system resource consumption, improving query performance, ensuring stable and efficient operation of the business, improving user experience, and helping the healthy development of enterprise business.

[0099] Corresponding to the above method, the present invention also provides a data storage automated analysis and management system, including:

[0100] A rule configuration module for constructing a migration strategy based on the real-time attributes of data;

[0101] A data evaluation module for evaluating the main business table in response to an analysis and management instruction to obtain an evaluation result;

[0102] A rule matching module for matching the evaluation result according to the migration strategy to determine the target migration data;

[0103] A data distribution module for performing a uniform distribution algorithm on the target migration data to obtain the sub-table number to be migrated;

[0104] A data migration module that migrates the target migration data to the target sub-table according to the sub-table serial number;

[0105] An operation monitoring module is used to continuously record management operation logs, monitor the status of the main business table and its sub-tables, and set the dynamic weight coefficients of the data in the current dynamic weight factor information according to the status of the main business table and its sub-tables and the real-time attributes of the data.

[0106] Corresponding to the above methods and systems, the present invention also provides a data storage automated analysis management device, including: a processor and a memory, the memory stores a computer program executable by the processor, and the processor implements the above methods when executing the computer program.

[0107] The technical advantages of the present invention are as follows:

[0108] Different from traditional database sharding and table partitioning that scatter data into multiple database tables and require frequent cross-database and cross-table operations during operations. The present invention takes the main business process table as the core, and the main process business maintains single-table operations, and only migrates and processes historical data. This method reduces the complexity of data management and retains the convenience of single-table operations, which can significantly improve development efficiency and system stability when dealing with daily business.

[0109] The present invention introduces an automated analysis mechanism that intelligently determines the migration data range based on multi-dimensional factors such as the time, status, and data volume of the data, and automatically executes the migration operation. Traditional database sharding and table partitioning usually require manual rule setting and manual implementation, which not only involves a large amount of work but also is prone to errors and is difficult to adapt to complex and changeable business scenarios.

[0110] Different from traditional database sharding and table partitioning that evenly split data according to certain rules, which may cause some active data to be scattered and affect query performance. The present invention screens data through precise migration rules, such as time dimension, business status, and data volume threshold, and only migrates historical data that does not affect the main business process, ensuring that the main business table always runs efficiently and improving the overall performance of the system.

[0111] Optimize query performance. In complex business scenarios such as the display of the post-management list and the joint query of multiple business tables after processing, traditional database sharding and table partitioning require complex association operations for queries due to the dispersion of data in multiple database tables, resulting in large performance losses. The integrity and singularity of the data in the main business table of this solution make the query operation simpler and more efficient, reducing the performance overhead of cross-database and cross-table queries.

[0112] This solution uses the modulo method of the hash value of the customer number combined with dynamic weight adjustment to distribute the migration data to all historical sub-tables, ensuring uniform data distribution and achieving dynamic data balance.

[0113] Different from traditional data sharding, which may break the correlation between data when splitting data, causing difficulties in data analysis and business processing. The data migrated in this solution is mainly historical data, and the integrity of recent data is still retained in the main business table, so that the data correlation can be better maintained, facilitating data analysis and the implementation of complex business logics.

[0114] The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. A method for automated analysis and management of data storage, characterized in that: It includes the following steps: S1. Construct a migration strategy based on the real-time attributes of data; S2. In response to the analysis management instruction, evaluate the main business table to obtain an evaluation result; S3. Match the evaluation result according to the migration strategy to determine the target migration data; S4. For the target migration data, execute the uniform distribution algorithm to obtain the sub-table serial numbers to be migrated; S5. According to the sub-table serial numbers, migrate the target migration data to the target sub-tables.

2. The data storage automated analysis and management method according to claim 1, wherein: The real-time attributes of data include: creation time, update time, business status, business attributes, and data volume; The evaluation result includes the creation time, update time, business status, business attributes, and data volume of each item of data in the main business table; The migration rules included in the migration strategy include: regular cleaning rules, time window screening rules, business completion determination rules, state transition trigger rules, main table threshold control rules, and sub-table balance rules.

3. A data storage automated analysis and management method according to claim 2, characterized in that: in In S1, after constructing the migration strategy, construct a decision tree model with each migration rule in the migration strategy as the fulcrum; The decision tree model is used to input the evaluation result, output the decision result, and determine the target migration data.

4. A method for automated analysis and management of data storage according to claim 3, characterized in that: in S3, after determining the target migration data, through the association rule mining algorithm, analyze the business association data of each item of target migration data, and determine whether the business association data all conforms to the migration strategy. If so, execute S4 for the target migration data. If not, remove this item of data from the target migration data.

5. A method for automated analysis and management of data storage according to claim 4, characterized in that: In S4, the uniform distribution algorithm includes modulo processing of hash values based on dynamic weight adjustment, and the process is as follows: Obtain the unique number included in each item of target migration data, determine the number of sub-tables corresponding to the main business table, obtain the current dynamic weight factor information, and find the dynamic weight coefficient matching the target migration data; Convert the format of the unique number of the target migration data to obtain a numerical number in a unified format; Perform an operation on the dynamic weight coefficient and the data number to obtain a final numerical number; For the final numerical number, perform hash modulo with the number of sub-tables as the basis to obtain the sub-table serial number; Determine the target sub-table to be migrated according to the sub-table serial number.

6. The automated analysis and management method for data storage according to claim 5, characterized in that: S3 In, after determining the target migration data, sort each item of target migration data according to the priority rule to obtain a data list; In S4, perform modulo processing of hash values on the target migration data in sequence according to the order of the data list.

7. A method for automated analysis and management of data storage according to claim 6, characterized in that: S5 In, read the target migration data from the main business table and perform integrity verification. If the data is complete, temporarily store the target migration data in the temporary storage area; Create a record space in the target sub-table corresponding to the sub-table serial number; Write the target data in the temporary storage area into the record space in the target sub-table, and during the writing process, perform consistency verification on the written target migration data and the original data in the main business table.

8. A method for automated analysis and management of data storage according to claim 7, characterized in that: in During the execution of S1-S5, continuously record the management operation logs, monitor the status of the main business table and its sub-tables, and set the dynamic weight coefficients of the data in the current dynamic weight factor information according to the status of the main business table and its sub-tables and the real-time attributes of the data; The management operation log includes the migration time, the amount of data, the location of the source data in the main business table, and the information of the target sub-table after migration.

9. A data storage automated analysis and management system, characterized in that: It includes: A rule configuration module for constructing a migration strategy based on the real-time attributes of data; A data evaluation module that evaluates the main business table in response to an analysis management instruction to obtain an evaluation result; A rule matching module that matches the evaluation result according to the migration strategy to determine the target migration data; A data distribution module that executes a uniform distribution algorithm on the target migration data to obtain the sub-table number to be migrated; A data migration module that migrates the target migration data to the target sub-table according to the sub-table number; An operation monitoring module for continuously recording the management operation log, monitoring the status of the main business table and its sub-tables, and setting the dynamic weight coefficients of the data in the current dynamic weight factor information according to the status of the main business table and its sub-tables and the real-time attributes of the data.

10. An automated analysis and management device for data storage, characterized in that: It includes: A processor and a memory, the memory stores a computer program executable by the processor, and when the processor executes the computer program, it implements the method described in any one of claims 1-8.

Images