Database access method and device based on data storage system

Abstract

The embodiment of the present application provides a database access method and device based on a data storage system, the method comprising: extracting scheduling features and synchronization features from historical database data through preset feature extraction rules; determining a corresponding scheduling model according to the scheduling features; determining a corresponding synchronization model according to the synchronization features; performing feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determining the corresponding fusion features, inputting the fusion features into a preset neural network model for model training, determining the corresponding difference persistence model, receiving a user request, inputting the user request into the difference persistence model, determining the corresponding optimal synchronization task sequence, and performing data synchronization operations on the database according to the optimal synchronization task sequence. The present application can improve the efficiency and consistency of data storage access.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a database access method and device based on a data storage system. Background Art

[0002] In traditional server-side development solutions, relational databases are the core of data storage and carry almost all business data in applications. However, with the expansion of business scale and the increase in the number of users, the performance bottlenecks and scalability issues faced by relational databases have become increasingly prominent. Especially in high-concurrency scenarios, frequent data read and write operations put tremendous pressure on database performance, resulting in a decrease in system response speed and a poor user experience.

[0003] To alleviate this problem, the industry generally uses cache technology to speed up database access. Among them, Redis (data storage system), as a high-performance memory data structure storage system, has become the first choice for cache technology due to its excellent read and write performance and rich data structure support. By storing hot data or query results in memory, Redis can significantly reduce the frequency of access to the database, increase the speed of data retrieval, and thus reduce the burden on the database.

[0004] However, although the traditional Redis cache and database combination solution can improve system performance to a certain extent, it still has some limitations. For example, data synchronization between cache and database, cache breakdown and avalanche phenomenon, and data consistency guarantee in high concurrency scenarios are all problems that need to be solved urgently, and cannot meet the requirements of efficient and accurate data storage and access. Summary of the invention

[0005] In view of the problems in the prior art, the present application provides a database access method and device based on a data storage system, which can improve the efficiency and consistency of data storage access.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a database access method based on a data storage system, comprising:

[0008] Acquire historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load features and preset priority features, and determine corresponding scheduling features; perform feature extraction operations on the preprocessed data according to preset change information features, and determine corresponding synchronization features, wherein the scheduling features are used to represent the optimal scheduling rules of the system, and the synchronization features are used to represent the data types that need to be changed synchronously;

[0009] Input the scheduling feature into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine the corresponding scheduling model; input the synchronization feature into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine the corresponding synchronization model;

[0010] Feature fusion is performed based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, corresponding fusion features are determined, the fusion features are input into a preset neural network model for model training, a corresponding difference persistence model is determined, a user request is received, the user request is input into the difference persistence model, a corresponding optimal synchronization task sequence is determined, and data synchronization operations are performed on the database according to the optimal synchronization task sequence.

[0011] Further, the performing of data preprocessing operation on the historical database data to determine corresponding preprocessed data includes:

[0012] Performing data cleaning operations on the historical database data to determine corresponding clean data;

[0013] A data normalization operation is performed on the clean data to determine corresponding preprocessed data.

[0014] Furthermore, the step of inputting the scheduling feature into a preset long short-term memory model, performing model training on the preset long short-term memory model according to preset parameter tuning rules, and determining a corresponding scheduling model includes:

[0015] Inputting the scheduling feature into a hidden layer of a preset long short-term memory model, adjusting the number of hidden layers and hidden units of the hidden layer according to a preset parameter tuning rule, and determining a corresponding optimal hidden layer structure;

[0016] An update operation is performed on the hidden layer of the preset long short-term memory model according to the optimal hidden layer structure to determine a corresponding scheduling model.

[0017] Furthermore, the step of inputting the synchronization feature into a preset clustering algorithm model, performing model training on the preset clustering algorithm model according to a preset clustering tuning rule, and determining a corresponding synchronization model includes:

[0018] Inputting the synchronization features into a preset clustering algorithm model, performing model training according to a preset number of cluster centers, and determining corresponding preliminary training results;

[0019] A parameter tuning operation is performed according to the silhouette coefficient and the preliminary training results to determine the corresponding synchronization model.

[0020] Further, the performing feature fusion according to the scheduling prediction feature output by the scheduling model and the synchronization prediction feature output by the synchronization model to determine the corresponding fusion feature includes:

[0021] Determining corresponding weight ratios according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model;

[0022] The scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model are weightedly summed according to the weight ratio to determine corresponding fusion features.

[0023] Furthermore, the step of inputting the fusion features into a preset neural network model for model training to determine a corresponding difference persistence model includes:

[0024] Input the fusion features into a preset multi-layer perceptron model for model training to determine the corresponding loss function;

[0025] According to the optimal loss function, a parameter tuning operation is performed on the preset multi-layer perceptron model to determine a corresponding difference persistence model.

[0026] Furthermore, the inputting the user request into the difference persistence model, determining a corresponding optimal synchronization task sequence, and performing a data synchronization operation on the database according to the optimal synchronization task sequence includes:

[0027] Input the user request and the preset system state into the difference persistence model to determine the corresponding priority execution tasks and synchronization data;

[0028] A data synchronization operation is performed on the database according to the priority execution task and the synchronization data.

[0029] In a second aspect, the present application provides a database access device based on a data storage system, comprising:

[0030] A feature extraction module is used to obtain historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load characteristics and preset priority characteristics, and determine corresponding scheduling characteristics; perform feature extraction operations on the preprocessed data according to preset change information characteristics, and determine corresponding synchronization characteristics, wherein the scheduling characteristics are used to represent the optimal scheduling rules of the system, and the synchronization characteristics are used to represent the data types that need to be changed synchronously;

[0031] The scheduling model and synchronization model determination module is used to input the scheduling features into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine the corresponding scheduling model; input the synchronization features into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine the corresponding synchronization model;

[0032] A differential synchronization module is used to perform feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into a preset neural network model for model training, determine the corresponding differential persistence model, receive user requests, input the user requests into the differential persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence.

[0033] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the database access method based on the data storage system when executing the program.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the database access method based on the data storage system.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the database access method based on the data storage system.

[0036] It can be seen from the above technical scheme that the present application provides a database access method and device based on a data storage system, which extracts scheduling features and synchronization features from historical database data through preset feature extraction rules; determines the corresponding scheduling model according to the scheduling features; determines the corresponding synchronization model according to the synchronization features; performs feature fusion based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determines the corresponding fusion features, inputs the fusion features into a preset neural network model for model training, determines the corresponding difference persistence model, receives user requests, inputs the user requests into the difference persistence model, determines the corresponding optimal synchronization task sequence, and performs data synchronization operations on the database according to the optimal synchronization task sequence, thereby improving the efficiency and consistency of data storage access. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is one of the flow charts of the database access method based on the data storage system in the embodiment of the present application;

[0039] Figure 2 This is a second flow chart of a database access method based on a data storage system in an embodiment of the present application;

[0040] Figure 3 This is a third flow chart of a database access method based on a data storage system in an embodiment of the present application;

[0041] Figure 4 This is a fourth flow chart of a database access method based on a data storage system in an embodiment of the present application;

[0042] Figure 5 This is a fifth flow chart of a database access method based on a data storage system in an embodiment of the present application;

[0043] Figure 6 This is a sixth flow chart of a database access method based on a data storage system in an embodiment of the present application;

[0044] Figure 7 This is the seventh flow chart of the database access method based on the data storage system in the embodiment of the present application;

[0045] Figure 8 A structural diagram of a database access device based on a data storage system in an embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0047] Reference numerals:

[0048] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0051] Considering the limitations of data synchronization faced by the combination of Redis cache and database, data consistency and system performance may be affected. The present application provides a database access method and device based on a data storage system, which extracts scheduling features and synchronization features from historical database data through preset feature extraction rules; determines the corresponding scheduling model according to the scheduling features; determines the corresponding synchronization model according to the synchronization features; performs feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determines the corresponding fusion features, inputs the fusion features into a preset neural network model for model training, determines the corresponding difference persistence model, receives user requests, inputs the user requests into the difference persistence model, determines the corresponding optimal synchronization task sequence, and performs data synchronization operations on the database according to the optimal synchronization task sequence, thereby improving the efficiency and consistency of data storage access.

[0052] In order to improve the efficiency and consistency of data storage access, the present application provides an embodiment of a database access method based on a data storage system, see Figure 1 The database access method based on the data storage system specifically includes the following contents:

[0053] Step S101: Acquire historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load features and preset priority features, and determine corresponding scheduling features; perform feature extraction operations on the preprocessed data according to preset change information features, and determine corresponding synchronization features, wherein the scheduling features are used to represent the optimal scheduling rules of the system, and the synchronization features are used to represent the data types that need to be changed synchronously;

[0054] Optionally, this solution is used as a whole to solve the data synchronization problem in Redis cluster and database storage access. Redis is used as the truth of data to save the latest and hottest data, and the Redis cluster data is synchronized with the database, which can solve the problem of data inconsistency. Furthermore, in data synchronization, the existing data synchronization method is based on fixed time synchronization. This solution further uses intelligent dual difference persistence to effectively solve the problem of low data synchronization efficiency in data storage access.

[0055] Specifically, persistence is the concept of persisting changed data in a database.

[0056] Specifically, dual differential persistence means that data is prioritized and stored differentially according to system conditions, and differential synchronization is performed according to the type of data change.

[0057] It is understandable that differential persistence is an important technical means to ensure data consistency, improve data processing efficiency and optimize resource utilization.

[0058] Optionally, in this embodiment, the purpose of step S101 is to lay a data foundation for subsequent difference persistence. Specifically, the scheduling features and synchronization features are extracted so that the subsequent model can perform optimal data scheduling and data synchronization based on the extracted scheduling features and synchronization features.

[0059] Optionally, in this embodiment, the technical means adopted in this step is to collect historical system operation data, and collect scheduling features and synchronization features respectively according to targeted feature extraction technology, so as to train the scheduling model and the synchronization model respectively.

[0060] Optionally, in this embodiment, the historical database data includes but is not limited to system monitoring data, data change logs, and user access data.

[0061] Optionally, in this embodiment, the optimal scheduling rule is to set priorities for different persistence tasks according to the importance and access frequency of the data, and dynamically adjust the priority and execution time of the persistence task according to the current load status of the system. For example, for hot data that users frequently access, a higher synchronization priority can be set to ensure that this data can be persisted to the database first.

[0062] Optionally, in this embodiment, the scheduling characteristics include system monitoring data and user access data.

[0063] Specifically, system monitoring data includes system performance indicators such as CPU usage, memory usage, network bandwidth, etc. User access data records the user's access frequency and access mode to data to evaluate the importance and popularity of data.

[0064] Optionally, in this embodiment, the type of data that is synchronized is changed by using the Redis change log or the timestamp, version number and other information in the database to identify the changed data and classify the data, and only synchronize the changed data.

[0065] Optionally, in this embodiment, the synchronization feature includes a data change log.

[0066] Specifically, the data change log records the changes in the data in Redis, including the change type, change time, data before and after the change, etc.

[0067] Optionally, in this embodiment, data preprocessing operations are performed on the historical database data to determine corresponding preprocessed data. This involves cleaning and standardizing monitoring data, change logs, and user access data. Cleaning and standardization ensure the quality and availability of data, and provide a solid foundation for subsequent data analysis and model training.

[0068] Specifically, in the data cleaning stage, missing value processing is first performed by supplementing the missing data through deletion and filling. Secondly, outlier detection is performed and outliers are processed through visualization. Finally, the data is converted into a format suitable for analysis, such as converting date strings into date types and converting text labels into numerical codes.

[0069] More specifically, the visualization method includes using visualization tools such as box plots and scatter plots to delete or replace outliers.

[0070] Specifically, in the data standardization stage, normalization is used to standardize the data. Robust standardization can be further used to use the median and interquartile range to reduce the impact of outliers.

[0071] Optionally, in this embodiment, a feature extraction operation is performed on the pre-processed data according to preset system load features and preset priority features to determine corresponding scheduling features.

[0072] Specifically, the scheduling features are used to help the machine learn the priority features of the data in the user request and determine the execution time based on the system resource conditions. Therefore, the scheduling features are obtained by extracting features using the preset system load feature rules and the preset priority feature rules. This can effectively help the model predict the task priority and execution time according to the system load conditions.

[0073] Specifically, the system load feature rules are preset to calculate the statistical indicator features such as the average CPU usage rate and the maximum memory usage rate per hour, such as the average value, maximum value, minimum value, and variance. At the same time, the daily and weekly trends of network traffic are extracted as its time series features. The priority feature rules are preset, and the user access frequency and pattern are analyzed using TF-IDF. Doc2Vec is used to convert text data into vector form to more accurately identify user access patterns and extract the optimal priority features for model training.

[0074] Optionally, in this embodiment, a feature extraction operation is performed on the pre-processed data according to preset change information features to determine corresponding synchronization features.

[0075] Specifically, the synchronization feature is used to help the machine learn the data type of the changed data for data synchronization judgment, including full data, incremental data, etc., so as to synchronize only the data that has changed since the last synchronization. In certain cases, such as system initialization or when the amount of data changes is extremely large, the algorithm will use the full synchronization method; in daily operation, the incremental synchronization method is mainly used, using the Redis change log or the timestamp, version number and other information in the database to achieve incremental synchronization, reducing unnecessary synchronization operations and data transmission volume.

[0076] Therefore, using preset change information feature rules to extract features and obtain synchronization features can effectively help the model optimize the synchronization strategy according to the data type of the change data.

[0077] Specifically, the preset change information features include data version or timestamp, change log, and synchronization record. When extracting features, these preset change information features will be analyzed and used to determine which data needs to be synchronized and how to arrange synchronization operations to maximize efficiency and minimize the impact on system performance. In this way, "synchronization features" that can guide data synchronization operations can be constructed, and these features will be used to train models to predict and optimize the data synchronization process.

[0078] It can be understood that after the above step S101, this embodiment realizes the extraction of scheduling features and synchronization features, so that in the subsequent model training process, the scheduling model can judge the data priority in the user instruction according to the scheduling features and prioritize the data according to the system load prediction to achieve optimal scheduling. The synchronization model can judge the data type of the changed data according to the synchronization features and predict the optimal synchronization strategy.

[0079] Step S102: input the scheduling feature into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine a corresponding scheduling model; input the synchronization feature into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine a corresponding synchronization model;

[0080] Optionally, in this embodiment, the scheduling characteristics are input into a preset long short-term memory model, and the preset long short-term memory model is trained according to preset parameter tuning rules to determine a corresponding scheduling model.

[0081] Optionally, the scheduling model selects a long short-term memory network (LSTM) sequence model to capture the timing characteristics of system performance indicators and combine it with user access data to predict task priority and execution time.

[0082] Optionally, preset parameter tuning rules, use system monitoring data and user access data to train the intelligent scheduling model, and adjust the model parameters so that the model can accurately predict task priority and execution time.

[0083] Specifically, adjust the number of hidden layers, number of hidden units, learning rate and other parameters of LSTM to optimize the prediction performance of the model. First, try the number of hidden layers from 1 to 3, because more layers may increase the complexity of the model and the risk of overfitting. Then, gradually increase the number of layers and observe the performance of the model on the validation set. If the performance improves significantly, continue to increase; if the performance starts to decline or there are signs of overfitting, stop increasing.

[0084] Secondly, the number of hidden units should be selected as a starting point according to the complexity of the problem and the dimension of the input data, such as 64, 128 or 256. The number of hidden units can be adjusted within a certain range through grid search or random search. Too many hidden units may lead to overfitting and increased computational cost.

[0085] Next, in order to avoid the problem of gradient explosion or disappearance in the early stage of training, the learning rate is selected as 0.001. Subsequently, the learning rate decay strategy or dynamic adjustment of the learning rate is used to adjust the learning rate to find the optimal learning rate.

[0086] Finally, these adjusted optimal parameters are used to update the long short-term memory model to obtain the scheduling model. The scheduling model can be used to evaluate the priority of user tasks and predict the optimal task execution time based on the current system status.

[0087] Optionally, in this embodiment, the synchronization feature is input into a preset clustering algorithm model, and the preset clustering algorithm model is trained according to a preset clustering tuning rule to determine a corresponding synchronization model.

[0088] Optionally, the synchronization model selects a clustering algorithm model (K-means algorithm), which can help identify and classify the data that needs to be synchronized.

[0089] Optionally, a cluster tuning rule is preset to improve the classification or clustering accuracy of the model. Specifically, first, the number of cluster centers to be formed is determined based on the specific problem and data distribution, and the elbow rule is used to help make the selection in the determination process.

[0090] Next, the maximum number of iterations and tolerance for algorithm execution are set, and a reasonable maximum number of iterations and a small tolerance are selected. In the system data table update embodiment, the maximum number of iterations is selected as 300 and the tolerance is selected as 1e-4.

[0091] Then, the model is trained based on the synchronization features, and indicators such as the silhouette coefficient and Calinski-Harabasz index are used to evaluate the clustering effect. If the algorithm converges too quickly (that is, the number of iterations is much less than the maximum number of iterations), consider reducing the tolerance to allow for finer convergence. If the clustering effect is not ideal, it is necessary to reconsider the choice of the number of cluster centers or try different initialization methods (such as k-means++). It is understandable that the most important thing in the model training process is to provide an evaluation standard for model optimization, and the silhouette coefficient can quantify the changes in model performance and enable R&D personnel to make adjustment decisions accordingly.

[0092] Finally, these adjusted optimal parameters are used to update the clustering algorithm model to obtain the synchronization model. Using the synchronization model, we can identify the data that needs to be synchronized and the data that does not need to be synchronized, and further classify and analyze the data that needs to be synchronized, identify incremental data or full data, and select different data synchronization strategies for data synchronization according to different types of changed data.

[0093] For example, suppose there are two data tables A and B in the system, where table A is a user information table with a low update frequency, and table B is an order information table with a high update frequency. In one day, a large number of changes occurred in the order information table B, while the changes in the user information table A were relatively small.

[0094] Scheduling model: Based on system monitoring data (such as the current low system load) and user access data (such as the access frequency of order information table B is much higher than that of user information table A), the intelligent scheduling model predicts that now is the best time to execute the persistence task of order information table B and sets its priority to the highest.

[0095] Synchronization model: By scanning the change log in Redis or checking the timestamp / version number information in the database, the differential synchronization model identifies the incremental data in the order information table B and outputs the classification results of these incremental data. Subsequently, according to the data classification results, the corresponding synchronization strategy is formulated, such as synchronizing incremental data first, synchronizing full data in batches, etc.

[0096] Step S103: Perform feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into a preset neural network model for model training, determine the corresponding difference persistence model, receive user requests, input the user requests into the difference persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence.

[0097] Optionally, in this embodiment, the purpose of this embodiment is to obtain a difference persistence model for automatically performing optimal double difference persistence according to the user's task request.

[0098] Optionally, in this embodiment, the technical means adopted in this embodiment is to fuse the outputs of the scheduling model and the synchronization model obtained in step S102, and use the fused features to train the neural network model so that the neural network model learns the features of the above two models.

[0099] Optionally, in this embodiment, feature fusion is performed based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model to determine corresponding fusion features.

[0100] Specifically, as described in step S102, after the scheduling model is trained with scheduling features, the system monitoring data and task characteristics are input into the intelligent scheduling model, and the model calculates the priority and execution time of each task. Priority considerations include, but are not limited to, the urgency of the task, the importance of the data, the current load of the system, etc. As described in step S102, after the synchronization model is trained, the synchronization model is used to classify the detected changed data. The classification basis includes, but is not limited to, data change type (such as incremental data, full data), data importance, change frequency, etc.

[0101] Next, key features are extracted from the prediction features output by the intelligent scheduling model and the synchronization model. These features include, but are not limited to, the urgency of the task, the type of data (incremental / full), the size of the data, the synchronization history, etc. At the same time, for the prediction features output by the synchronization model, synchronization strategy tags are further added to select synchronization strategies based on the data type. It can be understood that the classification model in the above S102 step is used to classify user requests, and synchronization tags are subsequently added to help the subsequent difference persistence model to automatically output classification strategies.

[0102] Optionally, feature fusion, first, based on the importance of the feature, the degree of influence on the final decision and the possible business logic, assign corresponding weights to each feature. The process of determining the weight uses the information gain algorithm to assist expert experience scoring, evaluates the importance of each feature, and assigns weights based on the importance. This can further ensure that the feature set used to train the model has the corresponding functionality.

[0103] Next, the output features of the intelligent scheduling model and the differentiated synchronization model are weighted and summed according to the determined weights. For numerical features, weighted sum is directly performed. For non-numerical features (such as classification labels), they are first encoded (such as one-hot encoding, label encoding, etc.) and then weighted.

[0104] Then, the weighted sum result is used as the fusion feature, together with other necessary features (such as timestamp, user ID, etc.), to form a fusion feature set.

[0105] It can be understood that the above steps are building a fused feature set with multiple feature connections. These fused features will be used as input for subsequent differential persistence model training to optimize the scheduling and execution of data synchronization tasks.

[0106] Optionally, in this embodiment, the fusion features are input into a preset neural network model for model training to determine a corresponding difference persistence model.

[0107] Specifically, this step is a model training process, in which the above fusion features are input into the multi-layer perceptron model for model training and the model parameters are adjusted according to the loss function obtained to obtain the final difference persistence model, and the difference persistence can be performed according to user requests.

[0108] Specifically, during the training process, appropriate loss functions and evaluation indicators are designed to monitor the model training process. The loss function should be able to reflect the difference between the model prediction results and the actual optimal synchronization task sequence; the evaluation indicators should include accuracy, recall rate, F1 score, etc., to comprehensively evaluate the performance of the model.

[0109] Specifically, the training set in the fusion feature set is used for gradient boosting model training to obtain a minimized loss function, and the model is reversely parameterized according to the minimum loss function to obtain the final difference persistence model. It can be understood that the difference persistence model combines the functions of the scheduling model and the classification model, and because the classification synchronization strategy label is added in the feature fusion processing stage, the synchronization strategy can also be determined according to the data type classified by the synchronization model.

[0110] Specifically, after obtaining the difference persistence model, it is necessary to use evaluation indicators to evaluate the model performance, and select corresponding evaluation indicators for each task type. At the same time, multiple indicators are combined for weighted summation to use comprehensive indicators to evaluate model performance for parameter tuning.

[0111] More specifically, multiple indicators are weighted and summed, because the difference persistence model in this step needs to process both classification features and regression features. Therefore, different weights are assigned to different evaluation indicators to reflect their importance to the overall performance.

[0112] For example, a neural network model handles both classification and regression tasks. For classification tasks, we care about accuracy and F1 score; for regression tasks, we care about mean square error (MSE) and coefficient of determination (R²). We can assign different weights to these indicators and sum them up to get a comprehensive performance indicator.

[0113] First, we define the weight of each indicator. Assume that we assign the following weights to these indicators based on business requirements:

[0114] Classification accuracy: Weight = 0.3

[0115] Classification F1-score: Weight = 0.3

[0116] Regression Mean Squared Error (MSE): Weight = -0.2 (Since MSE is a loss function, we want it to be as small as possible, so give it a negative weight)

[0117] Regression Coefficient of Determination (R²): Weight = 0.6 (R² is better when it is closer to 1, so give it a higher positive weight)

[0118] Now, we calculate the weighted sum based on the weight of each indicator and their values. Suppose we have the following indicator values:

[0119] Accuracy = 0.9

[0120] F1 score = 0.85

[0121] MSE = 0.05

[0122] R² = 0.9

[0123] Then according to the weighted summation formula: comprehensive performance index = 0.3 Accuracy +0.3 F1 score -0.2 MSE+0.6 R 2

[0124] The comprehensive performance index is 0.88. This index is used to evaluate the performance of the model and further adjust the model.

[0125] Optionally, in this embodiment, the user request is input into the difference persistence model, a corresponding optimal synchronization task sequence is determined, and a data synchronization operation is performed on the database according to the optimal synchronization task sequence.

[0126] It is understandable that the above steps are all in the training phase of the differential persistence model. This step is the application phase of the differential persistence model. After the model is deployed, when a user request is received, the user request is directly input into the trained differential persistence model. The differential persistence model directly generates the optimal synchronization task sequence based on the current system state (obtained through the real-time monitoring system) and the specific requirements in the user request, using the previously learned mapping relationship.

[0127] Specifically, the priority of task execution is dynamically assigned based on the current state of the system (such as load and resource usage) and the characteristics of the tasks to be processed (such as data volume, importance, and urgency). At the same time, the synchronized data is classified and differentiated to identify the characteristics and changes of different data.

[0128] Next, the execution time is determined according to the priority of task execution, and an accurate and efficient synchronization strategy is formulated according to the synchronization data classification. The execution time and execution strategy are used to perform data synchronization operations.

[0129] It can be understood that the difference persistence model performs two difference identifications during operation. In the first identification, the task priority is identified according to the importance and urgency of the task, and the execution time is determined according to the priority and system load. For example, the user inputs task A and task B, and the prediction result is obtained according to the relationship between the task and the system load identified by the model, that is, task A must be executed immediately, while task B can wait for the system to be idle. At this time, task A and task B are sent to the corresponding task execution channels respectively, where the channel of task A is the priority execution channel and the channel of task B is the waiting execution channel. It is worth noting that the execution channel can be defined according to the business model. For example, the priority execution channel is a channel that needs to be executed in any state of the system, and the waiting execution channel can continue to define how to execute when the system load is at the upper limit, the system load is too heavy, the system load is medium, the system load is light, and the system is idle. The execution channel is defined according to the above situation, which does not affect the implementation of the embodiment of the present invention.

[0130] The second identification is to identify the data type based on the changed data and formulate a corresponding synchronization strategy. For example, if the user enters task A and task B, the system recognizes that A includes a1 and a2, where a1 is existing data and a2 is incremental data. The system then captures incremental changes in real time and formulates a strategy to synchronize them to the database immediately. The system recognizes that there is no existing data for all data in B, so it recognizes that the B data is full data. When a full change is detected, batch synchronization tasks are arranged according to the system load to avoid excessive impact on system performance.

[0131] It is worth noting that the above two identifications are two complementary but relatively independent processes, that is, the difference persistence model performs two identifications almost at the same time and derives the execution strategy based on the identification results. These two processes are closely coordinated and work together to improve the efficiency and accuracy of data persistence.

[0132] This example shows how this embodiment performs secondary difference identification on user data and formulates a persistence strategy based on the difference persistence model to improve the consistency and efficiency of data synchronization.

[0133] From the above description, it can be seen that the database access method based on the data storage system provided in the embodiment of the present application can extract scheduling features and synchronization features from historical database data through preset feature extraction rules; determine the corresponding scheduling model according to the scheduling features; determine the corresponding synchronization model according to the synchronization features; perform feature fusion based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into the preset neural network model for model training, determine the corresponding difference persistence model, receive user requests, input the user requests into the difference persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence, thereby improving the efficiency and consistency of data storage access.

[0134] In one embodiment of the database access method based on the data storage system of the present application, see Figure 2 , and can also include the following:

[0135] Step S201: performing a data cleaning operation on the historical database data to determine corresponding clean data;

[0136] Step S202: performing a data normalization operation on the cleaned data to determine corresponding pre-processed data.

[0137] Optionally, in this embodiment, the data processing in this step is the basic work for all subsequent work.

[0138] Optionally, in this embodiment, the historical database data includes but is not limited to system monitoring data, data change logs, and user access data.

[0139] Optionally, in this embodiment, data preprocessing operations are performed on the historical database data to determine corresponding preprocessed data. This involves cleaning and standardizing monitoring data, change logs, and user access data. Cleaning and standardization ensure the quality and availability of data, and provide a solid foundation for subsequent data analysis and model training.

[0140] Specifically, in the data cleaning stage, missing value processing is first performed by supplementing the missing data through deletion and filling. Secondly, outlier detection is performed and outliers are processed through visualization. Finally, the data is converted into a format suitable for analysis, such as converting date strings into date types and converting text labels into numerical codes.

[0141] More specifically, the visualization method includes using visualization tools such as box plots and scatter plots to delete or replace outliers.

[0142] Specifically, in the data standardization stage, normalization is used to standardize the data. Robust standardization can be further used to use the median and interquartile range to reduce the impact of outliers.

[0143] Through step S202, this embodiment preprocesses the historical database information to lay a solid data foundation for subsequent model training.

[0144] In one embodiment of the database access method based on the data storage system of the present application, see Figure 3 , and can also include the following:

[0145] Step S301: inputting the scheduling feature into the hidden layer of a preset long short-term memory model, adjusting the number of hidden layers and hidden units of the hidden layer according to a preset parameter tuning rule, and determining a corresponding optimal hidden layer structure;

[0146] Step S302: updating the hidden layer of the preset long short-term memory model according to the optimal hidden layer structure to determine the corresponding scheduling model.

[0147] Optionally, in this embodiment, the scheduling characteristics include system monitoring data and user access data.

[0148] Specifically, system monitoring data includes system performance indicators such as CPU usage, memory usage, network bandwidth, etc. User access data records the user's access frequency and access mode to data to evaluate the importance and popularity of data.

[0149] Optionally, in this embodiment, the scheduling model selects a long short-term memory network (LSTM) sequence model to capture the timing characteristics of system performance indicators, and combines user access data to predict task priority and execution time.

[0150] Optionally, preset parameter tuning rules, use system monitoring data and user access data to train the intelligent scheduling model, and adjust the model parameters so that the model can accurately predict task priority and execution time.

[0151] Specifically, adjust the number of hidden layers, number of hidden units, learning rate and other parameters of LSTM to optimize the prediction performance of the model. First, try the number of hidden layers from 1 to 3, because more layers may increase the complexity of the model and the risk of overfitting. Then, gradually increase the number of layers and observe the performance of the model on the validation set. If the performance improves significantly, continue to increase; if the performance starts to decline or there are signs of overfitting, stop increasing.

[0152] Secondly, the number of hidden units should be selected as a starting point according to the complexity of the problem and the dimension of the input data, such as 64, 128 or 256. The number of hidden units can be adjusted within a certain range through grid search or random search. Too many hidden units may lead to overfitting and increased computational cost.

[0153] Next, in order to avoid the problem of gradient explosion or disappearance in the early stage of training, the learning rate is selected as 0.001. Subsequently, the learning rate decay strategy or dynamic adjustment of the learning rate is used to adjust the learning rate to find the optimal learning rate.

[0154] Finally, these adjusted optimal parameters are used to update the long short-term memory model to obtain the scheduling model. The scheduling model can be used to evaluate the priority of user tasks and predict the optimal task execution time based on the current system status.

[0155] For example, suppose there are two data tables A and B in the system, where table A is a user information table with a low update frequency, and table B is an order information table with a high update frequency. In one day, a large number of changes occurred in the order information table B, while the changes in the user information table A were relatively small.

[0156] Scheduling model: Based on system monitoring data (such as the current low system load) and user access data (such as the access frequency of order information table B is much higher than that of user information table A), the intelligent scheduling model predicts that now is the best time to execute the persistence task of order information table B and sets its priority to the highest.

[0157] Through step S302, this embodiment obtains a scheduling model that can perform optimal scheduling according to user requests and system load, laying a foundation for subsequent difference persistence.

[0158] In one embodiment of the database access method based on the data storage system of the present application, see Figure 4 , and can also include the following:

[0159] Step S401: input the synchronization features into a preset clustering algorithm model, perform model training according to a preset number of cluster centers, and determine a corresponding preliminary training result;

[0160] Step S402: Perform parameter tuning operations according to the silhouette coefficient and the preliminary training results to determine the corresponding synchronization model.

[0161] Optionally, in this embodiment, the synchronization feature includes a data change log.

[0162] Specifically, the data change log records the changes in the data in Redis, including the change type, change time, data before and after the change, etc.

[0163] Optionally, the synchronization model selects a clustering algorithm model (K-means algorithm), which can help identify and classify the data that needs to be synchronized.

[0164] Optionally, a cluster tuning rule is preset to improve the classification or clustering accuracy of the model. Specifically, first, the number of cluster centers to be formed is determined based on the specific problem and data distribution, and the elbow rule is used to help make the selection in the determination process.

[0165] Next, the maximum number of iterations and tolerance for algorithm execution are set, and a reasonable maximum number of iterations and a small tolerance are selected. In the system data table update embodiment, the maximum number of iterations is selected as 300 and the tolerance is selected as 1e-4.

[0166] Then, the model is trained based on the synchronization features, and indicators such as the silhouette coefficient and Calinski-Harabasz index are used to evaluate the clustering effect. If the algorithm converges too quickly (that is, the number of iterations is much less than the maximum number of iterations), consider reducing the tolerance to allow for finer convergence. If the clustering effect is not ideal, it is necessary to reconsider the choice of the number of cluster centers or try different initialization methods (such as k-means++). It is understandable that the most important thing in the model training process is to provide an evaluation standard for model optimization, and the silhouette coefficient can quantify the changes in model performance and enable R&D personnel to make adjustment decisions accordingly.

[0167] Finally, these adjusted optimal parameters are used to update the clustering algorithm model to obtain the synchronization model. Using the synchronization model, we can identify the data that needs to be synchronized and the data that does not need to be synchronized, and further classify and analyze the data that needs to be synchronized, identify incremental data or full data, and select different data synchronization strategies for data synchronization according to different types of changed data.

[0168] For example, suppose there are two data tables A and B in the system, where table A is a user information table with a low update frequency, and table B is an order information table with a high update frequency. In one day, a large number of changes occurred in the order information table B, while the changes in the user information table A were relatively small.

[0169] Synchronization model: By scanning the change log in Redis or checking the timestamp / version number information in the database, the differential synchronization model identifies the incremental data in the order information table B and outputs the classification results of these incremental data. Subsequently, according to the data classification results, the corresponding synchronization strategy is formulated, such as synchronizing incremental data first, synchronizing full data in batches, etc.

[0170] Through step S402, this embodiment successfully obtains the synchronization model, and can classify the changed data according to the user's request, laying a foundation for the subsequent formulation of strategies for the changed data type.

[0171] In one embodiment of the database access method based on the data storage system of the present application, see Figure 5 , and can also include the following:

[0172] Step S501: determining corresponding weight ratios according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model;

[0173] Step S502: performing weighted summation on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model according to the weight ratio to determine corresponding fusion features.

[0174] Optionally, in this embodiment, feature fusion is performed based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model to determine corresponding fusion features.

[0175] Specifically, as described in step S102, after the scheduling model is trained with scheduling features, the system monitoring data and task characteristics are input into the intelligent scheduling model, and the model calculates the priority and execution time of each task. Priority considerations include, but are not limited to, the urgency of the task, the importance of the data, the current load of the system, etc. As described in step S102, after the synchronization model is trained, the synchronization model is used to classify the detected changed data. The classification basis includes, but is not limited to, data change type (such as incremental data, full data), data importance, change frequency, etc.

[0176] Next, key features are extracted from the prediction features output by the intelligent scheduling model and the synchronization model. These features include, but are not limited to, the urgency of the task, the type of data (incremental / full), the size of the data, the synchronization history, etc. At the same time, for the prediction features output by the synchronization model, synchronization strategy tags are further added to select synchronization strategies based on the data type. It can be understood that the classification model in the above S102 step is used to classify user requests, and synchronization tags are subsequently added to help the subsequent difference persistence model to automatically output classification strategies.

[0177] Optionally, feature fusion, first, based on the importance of the feature, the degree of influence on the final decision and the possible business logic, assign corresponding weights to each feature. The process of determining the weight uses the information gain algorithm to assist expert experience scoring, evaluates the importance of each feature, and assigns weights based on the importance. This can further ensure that the feature set used to train the model has the corresponding functionality.

[0178] Next, the output features of the intelligent scheduling model and the differentiated synchronization model are weighted and summed according to the determined weights. For numerical features, weighted sum is directly performed. For non-numerical features (such as classification labels), they are first encoded (such as one-hot encoding, label encoding, etc.) and then weighted.

[0179] Then, the weighted sum result is used as the fusion feature, together with other necessary features (such as timestamp, user ID, etc.), to form a fusion feature set.

[0180] It can be understood that the above steps are building a fused feature set with multiple feature connections. These fused features will be used as input for subsequent difference persistence model training to optimize the scheduling and execution of data synchronization tasks.

[0181] Through step S502, this embodiment successfully determines the fusion feature set, laying a foundation for subsequent training of the difference persistence model.

[0182] In one embodiment of the database access method based on the data storage system of the present application, see Figure 6 , and can also include the following:

[0183] Step S601: inputting the fusion features into a preset multi-layer perceptron model for model training to determine a corresponding loss function;

[0184] Step S602: performing parameter tuning operations on the preset multi-layer perceptron model according to the optimal loss function to determine a corresponding difference persistence model.

[0185] Optionally, in this embodiment, the fusion features are input into a preset neural network model for model training to determine a corresponding difference persistence model.

[0186] Specifically, this step is a model training process, in which the above fusion features are input into the multi-layer perceptron model for model training and the model parameters are adjusted according to the loss function obtained to obtain the final difference persistence model, and the difference persistence can be performed according to user requests.

[0187] Specifically, during the training process, appropriate loss functions and evaluation indicators are designed to monitor the model training process. The loss function should be able to reflect the difference between the model prediction results and the actual optimal synchronization task sequence; the evaluation indicators should include accuracy, recall rate, F1 score, etc., to comprehensively evaluate the performance of the model.

[0188] Specifically, the training set in the fusion feature set is used for gradient boosting model training to obtain a minimized loss function, and the model is reversely parameterized according to the minimum loss function to obtain the final difference persistence model. It can be understood that the difference persistence model combines the functions of the scheduling model and the classification model, and because the classification synchronization strategy label is added in the feature fusion processing stage, the synchronization strategy can also be determined according to the data type classified by the synchronization model.

[0189] Through step S602, this embodiment successfully obtains a difference persistence model based on the minimum training loss, laying a foundation for subsequently realizing dual difference persistence based on the model and improving the consistency and efficiency of data synchronization.

[0190] In one embodiment of the database access method based on the data storage system of the present application, see Figure 7 , and can also include the following:

[0191] Step S701: input the user request and the preset system state into the difference persistence model to determine the corresponding priority execution tasks and synchronization data;

[0192] Step S702: performing data synchronization operation on the database according to the priority execution task and the synchronization data.

[0193] This step is optional and is the application phase of the differential persistence model. After the model is deployed, when a user request is received, the user request is directly input into the trained differential persistence model. The differential persistence model directly generates the optimal synchronization task sequence based on the current system state (obtained through the real-time monitoring system) and the specific requirements in the user request, using the previously learned mapping relationship.

[0194] Specifically, the priority of task execution is dynamically assigned based on the current state of the system (such as load and resource usage) and the characteristics of the tasks to be processed (such as data volume, importance, and urgency). At the same time, the synchronized data is classified and differentiated to identify the characteristics and changes of different data.

[0195] Next, the execution time is determined according to the priority of task execution, and an accurate and efficient synchronization strategy is formulated according to the synchronization data classification. The execution time and execution strategy are used to perform data synchronization operations.

[0196] It can be understood that the difference persistence model performs two difference identifications during operation. In the first identification, the task priority is identified according to the importance and urgency of the task, and the execution time is determined according to the priority and system load. For example, the user inputs task A and task B, and the prediction result is obtained according to the relationship between the task and the system load identified by the model, that is, task A must be executed immediately, while task B can wait for the system to be idle. At this time, task A and task B are sent to the corresponding task execution channels respectively, where the channel of task A is the priority execution channel and the channel of task B is the waiting execution channel. It is worth noting that the execution channel can be defined according to the business model. For example, the priority execution channel is a channel that needs to be executed in any state of the system, and the waiting execution channel can continue to define how to execute when the system load is at the upper limit, the system load is too heavy, the system load is medium, the system load is light, and the system is idle. The execution channel is defined according to the above situation, which does not affect the implementation of the embodiment of the present invention.

[0197] The second identification is to identify the data type based on the changed data and formulate a corresponding synchronization strategy. For example, if the user enters task A and task B, the system recognizes that A includes a1 and a2, where a1 is existing data and a2 is incremental data. The system then captures incremental changes in real time and formulates a strategy to synchronize them to the database immediately. The system recognizes that there is no existing data for all data in B, so it recognizes that the B data is full data. When a full change is detected, batch synchronization tasks are arranged according to the system load to avoid excessive impact on system performance.

[0198] It is worth noting that the above two identifications are two complementary but relatively independent processes, that is, the difference persistence model performs two identifications almost at the same time and derives the execution strategy based on the identification results. These two processes are closely coordinated and work together to improve the efficiency and accuracy of data persistence.

[0199] Through step S702, this embodiment successfully demonstrates how to perform secondary difference identification on user data and formulate a persistence strategy based on the difference persistence model to improve the consistency and efficiency of data synchronization.

[0200] In order to improve the efficiency and consistency of data storage access, the present application provides an embodiment of a database access device based on a data storage system for implementing all or part of the contents of the database access method based on the data storage system, see Figure 8 The database access device based on the data storage system specifically includes the following contents:

[0201] The feature extraction module 10 is used to obtain historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load characteristics and preset priority characteristics, and determine corresponding scheduling characteristics; perform feature extraction operations on the preprocessed data according to preset change information characteristics, and determine corresponding synchronization characteristics, wherein the scheduling characteristics are used to represent the optimal scheduling rules of the system, and the synchronization characteristics are used to represent the data types that need to be changed synchronously;

[0202] The scheduling model and synchronization model determination module 20 is used to input the scheduling features into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine the corresponding scheduling model; input the synchronization features into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine the corresponding synchronization model;

[0203] The differential synchronization module 30 is used to perform feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into a preset neural network model for model training, determine the corresponding differential persistence model, receive user requests, input the user requests into the differential persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence.

[0204] From the above description, it can be seen that the database access device based on the data storage system provided in the embodiment of the present application can extract scheduling features and synchronization features from historical database data through preset feature extraction rules; determine the corresponding scheduling model according to the scheduling features; determine the corresponding synchronization model according to the synchronization features; perform feature fusion based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into the preset neural network model for model training, determine the corresponding difference persistence model, receive user requests, input the user requests into the difference persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence, thereby improving the efficiency and consistency of data storage access.

[0205] From a hardware perspective, in order to improve the efficiency and consistency of data storage access, the present application provides an embodiment of an electronic device for implementing all or part of the contents of the database access method based on the data storage system, and the electronic device specifically includes the following contents:

[0206] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the database access method based on the data storage system and the core business system, user terminal and related database and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the database access method based on the data storage system and the embodiment of the database access method based on the data storage system in the embodiment, and the contents thereof are merged here, and the repeated parts are not repeated.

[0207] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0208] In practical applications, part of the database access method based on the data storage system can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0209] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0210] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0211] In one embodiment, the database access method function based on the data storage system can be integrated into the central processor 9100. The central processor 9100 can be configured to perform the following control:

[0212] Step S101: Acquire historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load features and preset priority features, and determine corresponding scheduling features; perform feature extraction operations on the preprocessed data according to preset change information features, and determine corresponding synchronization features, wherein the scheduling features are used to represent the optimal scheduling rules of the system, and the synchronization features are used to represent the data types that need to be changed synchronously;

[0213] Step S102: input the scheduling feature into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine a corresponding scheduling model; input the synchronization feature into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine a corresponding synchronization model;

[0214] Step S103: Perform feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into a preset neural network model for model training, determine the corresponding difference persistence model, receive user requests, input the user requests into the difference persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence.

[0215] From the above description, it can be seen that the electronic device provided in the embodiment of the present application extracts scheduling features and synchronization features from historical database data through preset feature extraction rules; determines the corresponding scheduling model according to the scheduling features; determines the corresponding synchronization model according to the synchronization features; performs feature fusion based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determines the corresponding fusion features, inputs the fusion features into a preset neural network model for model training, determines the corresponding difference persistence model, receives user requests, inputs the user requests into the difference persistence model, determines the corresponding optimal synchronization task sequence, and performs data synchronization operations on the database according to the optimal synchronization task sequence, thereby improving the efficiency and consistency of data storage access.

[0216] In another embodiment, the database access method based on the data storage system can be configured separately from the central processor 9100. For example, the database access method based on the data storage system can be configured as a chip connected to the central processor 9100, and the function of the database access method based on the data storage system is realized through the control of the central processor.

[0217] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0218] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0219] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0220] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0221] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0222] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0223] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0224] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the machine through the microphone 9132, and the sound stored on the machine can be played through the speaker 9131.

[0225] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the database access method based on a data storage system in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the database access method based on a data storage system in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0226] Step S101: Acquire historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load features and preset priority features, and determine corresponding scheduling features; perform feature extraction operations on the preprocessed data according to preset change information features, and determine corresponding synchronization features, wherein the scheduling features are used to represent the optimal scheduling rules of the system, and the synchronization features are used to represent the data types that need to be changed synchronously;

[0227] Step S102: input the scheduling feature into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine a corresponding scheduling model; input the synchronization feature into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine a corresponding synchronization model;

[0228] Step S103: Perform feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into a preset neural network model for model training, determine the corresponding difference persistence model, receive user requests, input the user requests into the difference persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence.

[0229] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application extracts scheduling features and synchronization features from historical database data through preset feature extraction rules; determines the corresponding scheduling model according to the scheduling features; determines the corresponding synchronization model according to the synchronization features; performs feature fusion based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determines the corresponding fusion features, inputs the fusion features into a preset neural network model for model training, determines the corresponding difference persistence model, receives user requests, inputs the user requests into the difference persistence model, determines the corresponding optimal synchronization task sequence, and performs data synchronization operations on the database according to the optimal synchronization task sequence, thereby improving the efficiency and consistency of data storage access.

[0230] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the database access method based on the data storage system in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the database access method based on the data storage system are implemented. For example, the computer program / instruction implements the following steps:

[0231] Step S101: Acquire historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load features and preset priority features, and determine corresponding scheduling features; perform feature extraction operations on the preprocessed data according to preset change information features, and determine corresponding synchronization features, wherein the scheduling features are used to represent the optimal scheduling rules of the system, and the synchronization features are used to represent the data types that need to be changed synchronously;

[0232] Step S102: input the scheduling feature into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine a corresponding scheduling model; input the synchronization feature into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine a corresponding synchronization model;

[0233] Step S103: Perform feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into a preset neural network model for model training, determine the corresponding difference persistence model, receive user requests, input the user requests into the difference persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence.

[0234] From the above description, it can be seen that the computer program product provided in the embodiment of the present application extracts scheduling features and synchronization features from historical database data through preset feature extraction rules; determines the corresponding scheduling model according to the scheduling features; determines the corresponding synchronization model according to the synchronization features; performs feature fusion based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determines the corresponding fusion features, inputs the fusion features into a preset neural network model for model training, determines the corresponding difference persistence model, receives user requests, inputs the user requests into the difference persistence model, determines the corresponding optimal synchronization task sequence, and performs data synchronization operations on the database according to the optimal synchronization task sequence, thereby improving the efficiency and consistency of data storage access.

[0235] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0236] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0237] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0238] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0239] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A database access method based on a data storage system, characterized in that: Applied to a scheduling platform, the scheduling platform is in communication with a data storage system cluster and a database, the data storage system cluster is used to cache user requests, and the database is used to store user requests, the method includes: Acquire historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load features and preset priority features, and determine corresponding scheduling features; perform feature extraction operations on the preprocessed data according to preset change information features, and determine corresponding synchronization features, wherein the scheduling features are used to represent the optimal scheduling rules of the system, and the synchronization features are used to represent the data types that need to be changed synchronously; Input the scheduling feature into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine the corresponding scheduling model; input the synchronization feature into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine the corresponding synchronization model; Feature fusion is performed based on the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, corresponding fusion features are determined, the fusion features are input into a preset neural network model for model training, a corresponding difference persistence model is determined, a user request is received, the user request is input into the difference persistence model, a corresponding optimal synchronization task sequence is determined, and data synchronization operations are performed on the database according to the optimal synchronization task sequence.

2. The database access method based on the data storage system according to claim 1, characterized in that: The performing of data preprocessing operation on the historical database data to determine corresponding preprocessed data includes: Performing data cleaning operations on the historical database data to determine corresponding clean data; A data normalization operation is performed on the clean data to determine corresponding preprocessed data.

3. The database access method based on the data storage system according to claim 1, characterized in that: The step of inputting the scheduling feature into a preset long short-term memory model, performing model training on the preset long short-term memory model according to a preset parameter tuning rule, and determining a corresponding scheduling model includes: Inputting the scheduling feature into a hidden layer of a preset long short-term memory model, adjusting the number of hidden layers and hidden units of the hidden layer according to a preset parameter tuning rule, and determining a corresponding optimal hidden layer structure; An update operation is performed on the hidden layer of the preset long short-term memory model according to the optimal hidden layer structure to determine a corresponding scheduling model.

4. The database access method based on the data storage system according to claim 1, characterized in that: The step of inputting the synchronization feature into a preset clustering algorithm model, performing model training on the preset clustering algorithm model according to a preset clustering tuning rule, and determining a corresponding synchronization model includes: Inputting the synchronization features into a preset clustering algorithm model, performing model training according to a preset number of cluster centers, and determining corresponding preliminary training results; A parameter tuning operation is performed according to the silhouette coefficient and the preliminary training results to determine the corresponding synchronization model.

5. The database access method based on the data storage system according to claim 1, characterized in that: The step of fusing the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model to determine the corresponding fusion features includes: Determining corresponding weight ratios according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model; The scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model are weightedly summed according to the weight ratio to determine corresponding fusion features.

6. The database access method based on the data storage system according to claim 1, characterized in that: The step of inputting the fusion features into a preset neural network model for model training to determine a corresponding difference persistence model includes: Input the fusion features into a preset multi-layer perceptron model for model training to determine the corresponding loss function; According to the optimal loss function, a parameter tuning operation is performed on the preset multi-layer perceptron model to determine a corresponding difference persistence model.

7. The database access method based on the data storage system according to claim 1, characterized in that: The step of inputting the user request into the difference persistence model, determining a corresponding optimal synchronization task sequence, and performing a data synchronization operation on the database according to the optimal synchronization task sequence includes: Input the user request and the preset system state into the difference persistence model to determine the corresponding priority execution tasks and synchronization data; A data synchronization operation is performed on the database according to the priority execution task and the synchronization data.

8. A database access device based on a data storage system, characterized in that: The device comprises: A feature extraction module is used to obtain historical database data, perform data preprocessing operations on the historical database data, determine corresponding preprocessed data, perform feature extraction operations on the preprocessed data according to preset system load characteristics and preset priority characteristics, and determine corresponding scheduling characteristics; perform feature extraction operations on the preprocessed data according to preset change information characteristics, and determine corresponding synchronization characteristics, wherein the scheduling characteristics are used to represent the optimal scheduling rules of the system, and the synchronization characteristics are used to represent the data types that need to be changed synchronously; The scheduling model and synchronization model determination module is used to input the scheduling features into a preset long short-term memory model, perform model training on the preset long short-term memory model according to preset parameter tuning rules, and determine the corresponding scheduling model; input the synchronization features into a preset clustering algorithm model, perform model training on the preset clustering algorithm model according to preset clustering tuning rules, and determine the corresponding synchronization model; A differential synchronization module is used to perform feature fusion according to the scheduling prediction features output by the scheduling model and the synchronization prediction features output by the synchronization model, determine the corresponding fusion features, input the fusion features into a preset neural network model for model training, determine the corresponding differential persistence model, receive user requests, input the user requests into the differential persistence model, determine the corresponding optimal synchronization task sequence, and perform data synchronization operations on the database according to the optimal synchronization task sequence.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the database access method based on the data storage system according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the database access method based on the data storage system described in any one of claims 1 to 7 are implemented.

Images