Database processing method and system based on message queue and synchronous processing

Abstract

The invention relates to the technical field of data processing, and discloses a database processing method and system based on a message queue and synchronous processing, and the method comprises the steps: calling an API of an Active MQ to insert data change information when data change occurs in a main database of a database; collecting data change information to determine the duration of a message queue, and collecting all data change information to construct a message data chain; obtaining importance data of each piece of data change information, splitting the message data chain and combining the message data chain into a new message data chain; extracting a feature vector of each data change message in the preferential processing data chain to determine a processing time length, and obtaining a final message data chain according to the processing time length; and adding the final message data chain into a List data structure of Redis, and obtaining a distributed lock according to the sequence of the final message data chain to carry out message processing. According to the method and the device, the message processing strategy is dynamically adjusted according to the service requirement, the data synchronization efficiency and accuracy are improved, and data synchronization can be realized in a high-concurrency environment.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a database processing method and system based on message queue and synchronous processing. Background Art

[0002] With the rapid development of information technology, enterprises rely on real-time processing and synchronization of large-scale data in their daily operations. Especially in database systems, real-time update and synchronization of data have a significant impact on the operational efficiency, decision support, and customer experience of enterprises. Currently, most enterprises adopt a database master-slave synchronization architecture, in which the master database is responsible for data writing operations and the slave database is responsible for data reading and backup. Under this architecture, data changes in the master database need to be synchronized to each slave database in a timely manner to ensure data consistency and reliability.

[0003] However, in actual applications, database master-slave synchronization faces some problems. For example, traditional database synchronization methods fail to effectively improve real-time performance, resulting in slow processing speed in high-concurrency scenarios. In multi-consumer concurrent processing scenarios, the processing order of message queues is easily confused, resulting in data inconsistency or synchronization failure, especially when the load is high, and the coordination between consumers becomes a bottleneck. Most current message processing uses the first-in-first-out (FIFO) principle, which cannot dynamically adjust the priority of messages according to business needs, resulting in some important data changes not being responded to in a timely manner, affecting the normal operation of the business.

[0004] Therefore, it is necessary to design a database processing method and system based on message queues and synchronous processing to solve the problems existing in the current technology. Summary of the invention

[0005] In view of this, the present invention proposes a database processing method based on message queue and synchronization processing, aiming to solve the problems existing in the current database master-slave synchronization technology, such as chaotic message queue processing resulting in poor backup effect, low backup speed and lack of dynamic adjustment of message priority.

[0006] In one aspect, the present invention provides a database processing method based on message queue and synchronization processing, comprising:

[0007] Collect data information from the database master library, and when data changes occur in the database master library, call the API of Active MQ to insert data change information;

[0008] Collect the data change information and determine the message queue duration according to the change information, collect all data change information within the message queue duration, and build a message data chain;

[0009] Using a neural network model to analyze each data change information in the message data chain, obtain importance data of each data change information, split the message data chain according to the importance data and combine it into a new message data chain, wherein the new message data chain includes a priority processing data chain and a non-priority processing data chain;

[0010] Extracting a characteristic vector of each data change message in the priority processing data chain, determining a processing time of each data change message according to the characteristic vector, sorting the priority processing data chains according to the processing time, and obtaining a final message data chain;

[0011] The final message data chain is added to the List data structure of Redis, and the distributed lock is obtained in the order of the final message data chain for message processing.

[0012] Further, when collecting the data change information and determining the message queue duration according to the change information, it includes:

[0013] Comparing the data change amount of the data change message with the first preset change amount and the second preset change amount respectively, and determining the message queue duration according to the comparison result, wherein the first preset change amount is smaller than the second preset change amount;

[0014] When the data change is less than or equal to the first preset change, the message queue duration is determined to be the first preset duration; when the data change is greater than the first preset change and less than or equal to the second preset change, the message queue duration is determined to be the second preset duration; when the data change is greater than the second preset change, the message queue duration is determined to be the third preset duration; the first preset duration is greater than the second preset duration, and the second preset duration is greater than the third preset duration.

[0015] Furthermore, when each data change information in the message data chain is analyzed using a neural network model to obtain importance data of each data change information, it includes:

[0016] Acquire historical data change information, and construct a historical data set based on the historical data change information;

[0017] Sampling the historical data set according to a preset ratio to obtain a training subset and a test subset;

[0018] A pre-selected neural network model is obtained, and the neural network model is iteratively trained according to the training subset, the iteratively trained neural network model is evaluated according to the test subset, and a final neural network model is obtained.

[0019] Furthermore, when each data change information in the message data chain is analyzed using a neural network model to obtain importance data of each data change information, it also includes:

[0020] Determine whether to stop iterative training based on the evaluation value;

[0021] If the evaluation value of the neural network model after the current iterative training is less than the evaluation value of the neural network model after the previous iterative training, the amplitude of the change of the neural network model in the gradient direction is reduced, and iterative training is continued until the preset number of iterations is reached;

[0022] If the evaluation value of the neural network model after the current iterative training is greater than or equal to the evaluation value of the neural network model after the previous iterative training, the iterative training is stopped.

[0023] Further, when the message data chain is split and combined into a new message data chain according to the importance data, it includes:

[0024] Acquire an initial data chain, wherein the initial data chain includes a first initial data chain and a second initial data chain, wherein each initial data chain is provided with a plurality of data chain nodes and connection nodes;

[0025] Generate a link change mark for all data change information on the message data link that is greater than or equal to the importance threshold;

[0026] Transferring all data change information carrying the link change mark to the first initial data link to obtain the priority processing data link;

[0027] The remaining data change information on the message data link is transferred to the second initial data link to obtain the non-priority processing data link.

[0028] Further, the priority processing data chains are sorted according to the processing time to obtain the final message data chain, including:

[0029] Performing Z-score normalization processing on the feature vectors of all data change messages in the priority processing data chain, wherein the feature vectors include the amount of data change, the degree of data change across tables, and the frequency of change;

[0030] Determine the initial neighborhood radius through the k-distance graph and set MinPts to 6;

[0031] Scan all points and find the points whose number of points in the neighborhood is greater than or equal to MinPts as core points;

[0032] Starting from each core point, check the points in its neighborhood; if the point in the neighborhood is a core point, continue to expand the cluster; if the point in the neighborhood is a boundary point, add it to the current cluster; if a point is not in the neighborhood of any core point and cannot form a cluster with other points, it is marked as a noise point;

[0033] When the core point is less than the number of data change information in the priority processing data chain, it is determined that the same type of change data exists in the priority processing data chain;

[0034] When the core point is equal to the number of data change information in the priority processing data chain, it is determined that there is no similar change data in the priority processing data chain.

[0035] Further, the priority processing data chains are sorted according to the processing time to obtain the final message data chain, and the method further includes:

[0036] Determining the processing time of each data change information in the priority processing data chain;

[0037] When it is determined that there is no similar changed data in the priority processing data chain, the priority processing data chain is sorted from high to low according to the processing time to obtain a final message data chain;

[0038] When it is determined that similar changed data exist in the priority processing data chain, the total processing time of each similar changed data is obtained according to the processing time of each data change information in the similar changed data, and the priority processing data chain is sorted from high to low according to the total processing time and the processing time in the similar changed data from high to low to obtain the final message data chain.

[0039] Further, determining the processing time of each data change information in the priority processing data chain includes:

[0040] Compare the feature vector with the historical data set, and determine the processing time of each data change information according to the comparison result;

[0041] When there is data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, the historical processing time of the historical data corresponding to the maximum similarity is used as the processing time of the data change information;

[0042] When there is no data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, data in the historical data set whose similarity with the feature vector is greater than a% and less than the similarity threshold is selected to establish an analogy set, and the processing time of the data change information is determined according to the analogy set.

[0043] Further, determining the processing time of the data change information according to the analogy set includes:

[0044] determining a median of the historical processing durations in the analog set;

[0045] Extracting the historical processing durations that are greater than the median among all the historical processing durations to construct a first data set;

[0046] Extracting the historical processing durations that are shorter than the median among all the historical processing durations to construct a second data set;

[0047] Determining a processing time of the data change information according to the first data set and the second data set;

[0048]

[0049] Among them, T represents the processing time of data change information, N represents the number of historical processing times in the first data set, Ti represents the i-th historical processing time in the first data set, T0 represents the median of the historical processing time in the analogy set, M represents the number of historical processing times in the second data set, and Tj represents the j-th historical processing time in the second data set.

[0050] Compared with the prior art, the beneficial effect of the present invention is that by collecting the real-time data change information of the database master and inserting it into the message queue through Active MQ, the real-time changes of the data can be effectively captured. The message data chain is analyzed by using a neural network model to dynamically evaluate the priority and importance of each data change, thereby realizing intelligent message priority division. By distinguishing the data chain for priority processing from the data chain for non-priority processing, and optimizing the processing time and order of each message according to the feature vector, it is ensured that important data changes can be processed with priority, and the system processing efficiency and real-time performance are improved. Combined with the List data structure and distributed lock mechanism of Redis, the message order problem during concurrent processing of multiple consumers is solved, the order and consistency of message processing are ensured, and data synchronization failures caused by concurrent conflicts are avoided. The performance and stability of the backup system are improved, the message processing strategy is dynamically adjusted according to business needs, the efficiency and accuracy of data synchronization are improved, and it is ensured that enterprises can achieve efficient and reliable data synchronization in a high-concurrency environment.

[0051] On the other hand, the present application also provides a database processing system based on message queue and synchronous processing, which is used to apply the above-mentioned database processing method based on message queue and synchronous processing, including:

[0052] The collection unit is configured to collect data information of the database master library, and when data changes occur in the database master library, call the API of Active MQ to insert the data change information;

[0053] A sorting unit is configured to collect the data change information and determine the message queue duration according to the change information, collect all data change information within the message queue duration, and build a message data chain;

[0054] an analysis unit configured to analyze each data change information in the message data chain using a neural network model, obtain importance data of each data change information, split the message data chain according to the importance data, and combine the message data chain into a new message data chain, wherein the new message data chain includes a priority processing data chain and a non-priority processing data chain;

[0055] an identification unit configured to extract a feature vector of each of the data change messages in the priority processing data chain, determine a processing time of each of the data change messages according to the feature vector, sort the priority processing data chains according to the processing time, and obtain a final message data chain;

[0056] The processing unit is configured to add the final message data chain to the List data structure of Redis, and obtain the distributed lock according to the order of the final message data chain to perform message processing.

[0057] It is understandable that the above-mentioned database processing method and system based on message queue and synchronous processing have the same beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0059] Figure 1 A flowchart of a database processing method based on message queue and synchronization processing provided by an embodiment of the present invention;

[0060] Figure 2 A structural block diagram of a database processing system based on message queues and synchronous processing provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] In some embodiments of the present application, see Figure 1 As shown, a database processing method based on message queue and synchronous processing includes:

[0063] S100: Collect data information of the database master database. When data changes occur in the database master database, call the API of ActiveMQ to insert the data change information.

[0064] S200: Collect data change information and determine the message queue duration according to the change information, collect all data change information within the message queue duration, and build a message data chain.

[0065] S300: Analyze each data change information in the message data chain using a neural network model to obtain importance data of each data change information, split the message data chain according to the importance data and combine it into a new message data chain, wherein the new message data chain includes a priority processing data chain and a non-priority processing data chain.

[0066] S400: extracting a feature vector of each data change message in a priority processing data chain, determining a processing time of each data change message according to the feature vector, sorting the priority processing data chains according to the processing time, and obtaining a final message data chain.

[0067] S500: Add the final message data chain to the List data structure of Redis, and obtain the distributed lock according to the final message data chain sequence to process the message.

[0068] Specifically, S100 monitors the data changes in the database master database in real time and promptly captures any insert, update or delete operations that occur in the database. When the data in the master database changes, the Active MQ API is called to insert the change information into the message queue in the form of a message. Database triggers, log monitoring, or database change capture technologies can be used to monitor data changes in the database master database in real time, and send change messages to the message queue through the Active MQ API. Active MQ is an open source messaging middleware that allows applications to send and receive messages asynchronously through message queues. API refers to the interface and method set used to interact with the messaging middleware, usually including client and server operations, such as sending, receiving, and managing message queues.

[0069] In S200, the collected data change information is classified and the duration of the message queue (i.e., the data collection time window) is determined. Within this duration, all relevant data change information will be collected and organized into a message data chain. Ensure that the system can process and manage all relevant change information. By setting the duration of the message queue, it is ensured that the system obtains all change information within a specific time window and aggregates this information into a data chain in chronological order. Each change information will be queued in the message queue for processing. In S300, a neural network model is introduced to analyze each data change information and calculate its importance. Each change information will be evaluated for an importance data based on the data change position. Based on these importance data, the message data chain is split into a priority processing data chain and a non-priority processing data chain, and the priority processing data will be arranged for priority consumption. The neural network model can predict the business importance of data changes through training data. This process generates an importance score for each data change based on feature engineering, and performs splitting operations based on the score. In S400, the feature vectors of each data change message in the data chain are extracted for priority processing, and the processing time of each message is evaluated using these feature vectors, and the messages are sorted according to the processing time. The sorted data chain will be processed in priority order. The feature vector is a numerical description of each message, which includes factors that affect the processing time, such as the amount of data change, the degree of data change across tables, and the frequency of change. In S500, after all messages are sorted and prepared, the final message data chain will be added to the Redis List data structure to ensure that they are processed in order. Each message will be consumed only after obtaining a distributed lock, ensuring that only one consumer can process a specific message, thereby avoiding concurrency conflicts and disordered order. The Redis List data structure is used to store message queues arranged in order. Distributed locks (such as SETNX or Redlock based on Redis) ensure that when multiple consumers process messages at the same time, there will be no repeated consumption of messages or disordered consumption order.

[0070] It is understandable that by collecting real-time data changes from the database master and inserting them into the message queue, the real-time nature of data synchronization is guaranteed. By analyzing and evaluating the priority of data changes through a neural network model, the message data chain is intelligently divided and sorted, ensuring that important business data can be processed first, avoiding important data processing delays caused by the traditional FIFO method. The use of Redis List data structure and distributed lock technology ensures the order and consistency of messages during concurrent processing of multiple consumers, effectively avoiding concurrency conflicts and data inconsistency problems. Efficient message processing can be maintained in a high-concurrency environment, improving the real-time, accuracy and reliability of data synchronization, and adapting to data processing and synchronization scenarios under complex business needs.

[0071] In some embodiments of the present application, when collecting data change information and determining the message queue duration based on the change information, it includes: comparing the data change amount of the data change message with the first preset change amount and the second preset change amount respectively, determining the message queue duration based on the comparison result, and the first preset change amount is smaller than the second preset change amount.

[0072] Specifically, when the data change amount is less than or equal to the first preset change amount, the message queue duration is determined to be the first preset duration. When the data change amount is greater than the first preset change amount and less than or equal to the second preset change amount, the message queue duration is determined to be the second preset duration. When the data change amount is greater than the second preset change amount, the message queue duration is determined to be the third preset duration. The first preset duration is greater than the second preset duration, and the second preset duration is greater than the third preset duration.

[0073] It is understandable that flexibly adjusting the message queue duration according to the actual amount of data change, rather than using a fixed duration, avoids wasting resources when the amount of change is low or causing processing timeouts when the amount of change is high. By adjusting the queue duration according to the amount of change, small-scale changes can be processed quickly, and sufficient time is allocated for processing when large-scale changes occur, avoiding processing delays caused by insufficient queue duration. Larger data changes usually require more time and resources to process. Through reasonable queue duration allocation, large-scale data changes can be effectively responded to, the risk of concurrent conflicts can be reduced, and overall processing efficiency can be improved.

[0074] In some embodiments of the present application, a neural network model is used to analyze each data change information in a message data chain to obtain importance data of each data change information, including:

[0075] Obtain historical data change information and build a historical data set based on the historical data change information.

[0076] The historical data set is sampled according to a preset ratio to obtain a training subset and a test subset.

[0077] A pre-selected neural network model is obtained, and the neural network model is iteratively trained according to the training subset, the iteratively trained neural network model is evaluated according to the test subset, and a final neural network model is obtained.

[0078] In some embodiments of the present application, when a neural network model is used to analyze each data change information in a message data chain to obtain importance data of each data change information, it also includes:

[0079] Determine whether to stop iterative training based on the evaluation value.

[0080] If the evaluation value of the neural network model after the current iterative training is less than the evaluation value of the neural network model after the previous iterative training, the amplitude of the change of the neural network model in the gradient direction is reduced, and the iterative training is continued until the preset number of iterations is reached.

[0081] If the evaluation value of the neural network model after the current iterative training is greater than or equal to the evaluation value of the neural network model after the previous iterative training, the iterative training is stopped.

[0082] It is understandable that a large amount of historical data change information is needed to train the neural network model. This historical data change information includes insert, update, and delete operations in the database, including the time, type, content, and tables and fields involved in the change. By collecting and organizing these historical change data, a historical data set is constructed, which will serve as the training basis for the neural network. Collect historical data change information through database logs. The data comes from database change records, triggers, or application logs. Clean and organize the collected historical data, and convert useful information (such as change time, operation type, changed tables and fields) into a format that can be input into the neural network. Divide the data set into a training subset and a test subset. The training subset is used for model training, and the test subset is used to verify the accuracy of the model to ensure that the model does not overfit. According to the preset ratio (for example, 80% for training and 20% for testing), sample the training subset and test subset from the historical data set. Select a neural network model (such as a multilayer perceptron or a convolutional neural network), use the training subset to iteratively train the neural network model, and minimize the prediction error by gradually adjusting the parameters of the model. The training process will continue until the evaluation value of the model meets the preset stop condition. The training subset is used to perform forward and backward propagation on the neural network, and the weight of the network is gradually adjusted through the optimization algorithm (such as gradient descent method). The trained model is evaluated using the test subset, and the effect of the model is measured by evaluation indicators (such as accuracy, precision, recall, etc.). The evaluation value of the model will reflect the prediction ability of the current model. When the evaluation value continues to increase, it means that the model is gradually improving. If it is found that the evaluation value of the model after the current iteration training is lower than that of the previous one, it means that the model has problems of overfitting or overtraining. At this time, it is necessary to adjust the learning rate or change the training method. If the evaluation value of the current model is less than or equal to the evaluation value of the previous one, it means that the model has stabilized and the training is stopped. Otherwise, continue training according to the adjustment strategy until the preset maximum number of iterations is reached or the loss value converges. After the model training is completed, the neural network analyzes each piece of data change information and outputs an importance data. This importance data is a scalar value that indicates the relative importance of the data change information in database synchronization. The calculation of this importance data will be based on the location of the data change. For example, when the change information occurs in the key table or important field of the main database, it will be given a higher priority.

[0083] It is understandable that the neural network model analyzes the historical information of data changes and automatically identifies which data changes are more critical and urgent. The location of the data in the database is taken into account to ensure that the core business data can be processed first. The neural network model automatically learns and optimizes the priority evaluation based on historical data. This avoids the problem that manual rules cannot cover all scenarios. Through the adaptive training and adjustment of the neural network, the processing strategy can be automatically adjusted according to changes in business needs and database environment, improving the degree of automation and adaptability.

[0084] In some embodiments of the present application, when a message data chain is split and combined into a new message data chain according to importance data, it includes: obtaining an initial data chain, the initial data chain includes a first initial data chain and a second initial data chain, wherein each initial data chain is provided with a plurality of data chain nodes and connection nodes. All data change information on the message data chain that is greater than or equal to an importance threshold is generated into a chain change mark.

[0085] Specifically, all data change information carrying the link change mark is transferred to the first initial data link to obtain a priority processing data link. The remaining data change information on the message data link is transferred to the second initial data link to obtain a non-priority processing data link.

[0086] In some embodiments of the present application, priority processing data chains are sorted according to processing time to obtain the final message data chain, including: Z-score normalization of feature vectors of all data change messages in the priority processing data chain, the feature vectors including the amount of data change, the degree of data change across tables, and the frequency of change.

[0087] The initialization neighborhood radius is determined through the k-distance graph, and MinPts is determined to be 6.

[0088] Scan all points and find the points whose number of points in the neighborhood is greater than or equal to MinPts as core points.

[0089] Starting from each core point, check the points in its neighborhood. If the point in the neighborhood is a core point, continue to expand the cluster. If the point in the neighborhood is a boundary point, add it to the current cluster. If a point is not in the neighborhood of any core point and cannot form a cluster with other points, it is marked as a noise point.

[0090] Specifically, when the core point is less than the number of data change information in the priority processing data chain, it is determined that the priority processing data chain has the same type of change data. When the core point is equal to the number of data change information in the priority processing data chain, it is determined that the priority processing data chain does not have the same type of change data.

[0091] In some embodiments of the present application, when the priority processing data chain is sorted according to the processing time to obtain the final message data chain, it also includes: determining the processing time of each data change information in the priority processing data chain. When it is determined that there is no similar change data in the priority processing data chain, the priority processing data chain is sorted from high to low according to the processing time to obtain the final message data chain. When it is determined that there is similar change data in the priority processing data chain, the total processing time of each similar change data is obtained according to the processing time of each data change information in the similar change data, and the priority processing data chain is sorted from high to low according to the total processing time and the processing time in the similar change data is from high to low to obtain the final message data chain.

[0092] It is understandable that in the priority processing data chain, the processing time of data change information is used as the basis for sorting. In order to better sort and optimize this information, this information is standardized and evaluated according to feature vectors such as data change amount, data change cross-table degree and change frequency. The feature vector of each data change information in the priority processing data chain is Z-score standardized so that data change information with different characteristics can be compared on the same scale. Through the standardized feature vector, the information in the priority processing data chain will be sorted according to the processing time to ensure that the most important or most urgent data changes are processed first. Through the k-distance graph, the MinPts clustering algorithm is used to analyze the data change information in the priority processing data chain. Identify whether there is similar change data in the data in the priority processing data chain, that is, whether there is a group of similar data change information that needs to be processed together. Further optimize the processing order of data changes and reduce redundant processing. Determine whether each data change information is a core point by calculating the neighborhood radius of the data change information. When there is similar change data in the priority processing data chain, in addition to considering the processing time of each data change information, the total processing time of the same change data needs to be calculated. If the total duration of the same type of change data is longer, these data will be processed first, and the processing duration of each data will be arranged in descending order to achieve optimal processing.

[0093] It is understandable that by splitting the message data chain into priority processing data chain and non-priority processing data chain according to importance data and chain change mark, it can ensure that important data changes are processed first, and improve the response speed and accuracy of business core data synchronization. Through the sorting and processing mechanism, it can effectively reduce the delay problem caused by long data processing time or frequent data changes, and improve the overall efficiency and real-time performance of the synchronization system. By combining importance data, processing time and cluster analysis of similar data, the entire synchronization system is more adaptable to complex database environments and high-concurrency scenarios, and can flexibly adjust the processing strategy according to actual needs.

[0094] In some embodiments of the present application, determining the processing time for each data change information in the priority processing data chain includes: comparing the feature vector with the historical data set, and determining the processing time for each data change information according to the comparison result.

[0095] Specifically, when there is data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, the historical processing time of the historical data corresponding to the maximum similarity is used as the processing time of the data change information. When there is no data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, the data in the historical data set whose similarity with the feature vector is greater than a% and less than the similarity threshold is selected to establish an analogy set, and the processing time of the data change information is determined based on the analogy set.

[0096] In some embodiments of the present application, when determining the processing time of data change information according to the analog set, the method includes: determining the median of the historical processing time in the analog set. Extracting the historical processing time greater than the median from all the historical processing time to construct a first data set. Extracting the historical processing time less than the median from all the historical processing time to construct a second data set. Determining the processing time of the data change information according to the first data set and the second data set.

[0097]

[0098] Among them, T represents the processing time of data change information, N represents the number of historical processing times in the first data set, Ti represents the i-th historical processing time in the first data set, T0 represents the median of the historical processing time in the analogy set, M represents the number of historical processing times in the second data set, and Tj represents the j-th historical processing time in the second data set.

[0099] It is understandable that by comparing with historical data, the processing time of each data change is predicted, avoiding system delays or over-processing caused by overly simple predictions. Through dynamic adjustment of similarity thresholds and analogy sets, the prediction of processing time can be flexibly adjusted according to actual data changes to make it more in line with actual business needs. Through median analysis of historical data sets, processing resources can be efficiently allocated to ensure timely processing of important data without being slowed down by a large amount of small change information. By comparing feature vectors with historical data, errors caused by overly broad processing time estimates can be reduced, avoiding unnecessary processing delays or waste of resources.

[0100] In the above embodiment, by collecting the real-time data change information of the database master library and inserting it into the message queue through Active MQ, the real-time changes of the data are effectively captured. The message data chain is analyzed by using a neural network model to dynamically evaluate the priority and importance of each data change, thereby realizing intelligent message priority division. By distinguishing the data chain for priority processing from the data chain for non-priority processing, and optimizing the processing time and order of each message according to the feature vector, it is ensured that important data changes can be processed with priority, and the system processing efficiency and real-time performance are improved. Combined with the List data structure and distributed lock mechanism of Redis, the message order problem during concurrent processing of multiple consumers is solved, the order and consistency of message processing are ensured, and data synchronization failures caused by concurrent conflicts are avoided. The performance and stability of the backup system are improved, the message processing strategy is dynamically adjusted according to business needs, the efficiency and accuracy of data synchronization are improved, and it is ensured that enterprises can achieve efficient and reliable data synchronization in a high-concurrency environment.

[0101] In another preferred embodiment based on the above embodiment, refer to Figure 2 As shown, this embodiment provides a database processing system based on message queue and synchronous processing, which is applied to the above-mentioned database processing method based on message queue and synchronous processing, including:

[0102] The collection unit is configured to collect data information of the database master library. When data changes occur in the database master library, the API of Active MQ is called to insert the data change information.

[0103] The collating unit is configured to collect data change information and determine the message queue duration according to the change information, collect all data change information within the message queue duration, and build a message data chain;

[0104] an analysis unit configured to analyze each data change information in the message data chain using a neural network model, obtain importance data of each data change information, split the message data chain according to the importance data, and combine the message data chain into a new message data chain, wherein the new message data chain includes a priority processing data chain and a non-priority processing data chain;

[0105] an identification unit configured to extract a feature vector of each data change message in the priority processing data chain, determine a processing time of each data change message according to the feature vector, sort the priority processing data chains according to the processing time, and obtain a final message data chain;

[0106] The processing unit is configured to add the final message data chain to the List data structure of Redis, and obtain the distributed lock to process the message according to the order of the final message data chain.

[0107] It is understandable that by collecting real-time data change information from the database master and inserting it into the message queue through ActiveMQ, the real-time changes of data can be effectively captured. The message data chain is analyzed using a neural network model to dynamically evaluate the priority and importance of each data change, thereby realizing intelligent message priority division. By distinguishing the data chain for priority processing from the data chain for non-priority processing, and optimizing the processing time and order of each message according to the feature vector, it is ensured that important data changes can be processed first, and the system processing efficiency and real-time performance are improved. Combined with Redis's List data structure and distributed lock mechanism, the message order problem during concurrent processing of multiple consumers is solved, the order and consistency of message processing are ensured, and data synchronization failures caused by concurrent conflicts are avoided. The performance and stability of the backup system are improved, the message processing strategy is dynamically adjusted according to business needs, the efficiency and accuracy of data synchronization are improved, and the enterprise can achieve efficient and reliable data synchronization in a high-concurrency environment.

[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A database processing method based on message queue and synchronous processing, characterized in that: include: Collect data information from the database master library, and when data changes occur in the database master library, call the API of Active MQ to insert data change information; Collect the data change information and determine the message queue duration according to the change information, collect all data change information within the message queue duration, and build a message data chain; Using a neural network model to analyze each data change information in the message data chain, obtain importance data of each data change information, split the message data chain according to the importance data and combine it into a new message data chain, wherein the new message data chain includes a priority processing data chain and a non-priority processing data chain; Extracting a characteristic vector of each data change message in the priority processing data chain, determining a processing time of each data change message according to the characteristic vector, sorting the priority processing data chains according to the processing time, and obtaining a final message data chain; The final message data chain is added to the List data structure of Redis, and the distributed lock is obtained in the order of the final message data chain for message processing.

2. The database processing method based on message queue and synchronous processing according to claim 1 is characterized in that: Collecting the data change information and determining the message queue duration according to the change information includes: Comparing the data change amount of the data change message with the first preset change amount and the second preset change amount respectively, and determining the message queue duration according to the comparison result, wherein the first preset change amount is smaller than the second preset change amount; When the data change is less than or equal to the first preset change, the message queue duration is determined to be the first preset duration; when the data change is greater than the first preset change and less than or equal to the second preset change, the message queue duration is determined to be the second preset duration; when the data change is greater than the second preset change, the message queue duration is determined to be the third preset duration; the first preset duration is greater than the second preset duration, and the second preset duration is greater than the third preset duration.

3. The database processing method based on message queue and synchronous processing according to claim 1 is characterized in that: When each data change information in the message data chain is analyzed using a neural network model to obtain importance data of each data change information, it includes: Acquire historical data change information, and construct a historical data set based on the historical data change information; Sampling the historical data set according to a preset ratio to obtain a training subset and a test subset; A pre-selected neural network model is obtained, and the neural network model is iteratively trained according to the training subset, the iteratively trained neural network model is evaluated according to the test subset, and a final neural network model is obtained.

4. The database processing method based on message queue and synchronous processing according to claim 3 is characterized in that: When analyzing each data change information in the message data chain using a neural network model to obtain importance data of each data change information, it also includes: Determine whether to stop iterative training based on the evaluation value; If the evaluation value of the neural network model after the current iterative training is less than the evaluation value of the neural network model after the previous iterative training, the amplitude of the change of the neural network model in the gradient direction is reduced, and the iterative training is continued until the preset number of iterations is reached; If the evaluation value of the neural network model after the current iterative training is greater than or equal to the evaluation value of the neural network model after the previous iterative training, the iterative training is stopped.

5. The database processing method based on message queue and synchronous processing according to claim 3 is characterized in that: When the message data chain is split and combined into a new message data chain according to the importance data, it includes: Acquire an initial data chain, wherein the initial data chain includes a first initial data chain and a second initial data chain, wherein each initial data chain is provided with a plurality of data chain nodes and connection nodes; Generate a link change mark for all data change information on the message data link that is greater than or equal to the importance threshold; Transferring all data change information carrying the link change mark to the first initial data link to obtain the priority processing data link; The remaining data change information on the message data link is transferred to the second initial data link to obtain the non-priority processing data link.

6. The database processing method based on message queue and synchronous processing according to claim 1 is characterized in that: The priority processing data chains are sorted according to the processing time to obtain the final message data chain, including: Performing Z-score normalization processing on the feature vectors of all data change messages in the priority processing data chain, wherein the feature vectors include the amount of data change, the degree of data change across tables, and the frequency of change; Determine the initial neighborhood radius through the k-distance graph and set MinPts to 6; Scan all points and find the points whose number of points in the neighborhood is greater than or equal to MinPts as core points; Starting from each core point, check the points in its neighborhood; if the point in the neighborhood is a core point, continue to expand the cluster; if the point in the neighborhood is a boundary point, add it to the current cluster; if a point is not in the neighborhood of any core point and cannot form a cluster with other points, it is marked as a noise point; When the core point is less than the number of data change information in the priority processing data chain, it is determined that the same type of change data exists in the priority processing data chain; When the core point is equal to the number of data change information in the priority processing data chain, it is determined that there is no similar change data in the priority processing data chain.

7. The database processing method based on message queue and synchronous processing according to claim 6 is characterized in that: The priority processing data chains are sorted according to the processing time to obtain the final message data chain, and the method further includes: Determining the processing time of each data change information in the priority processing data chain; When it is determined that there is no similar changed data in the priority processing data chain, the priority processing data chain is sorted from high to low according to the processing time to obtain a final message data chain; When it is determined that similar changed data exist in the priority processing data chain, the total processing time of each similar changed data is obtained according to the processing time of each data change information in the similar changed data, and the priority processing data chain is sorted from high to low according to the total processing time and the processing time in the similar changed data from high to low to obtain the final message data chain.

8. The database processing method based on message queue and synchronous processing according to claim 7 is characterized in that: Determining the processing time of each data change information in the priority processing data chain includes: Compare the feature vector with the historical data set, and determine the processing time of each data change information according to the comparison result; When there is data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, the historical processing time of the historical data corresponding to the maximum similarity is used as the processing time of the data change information; When there is no data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, data in the historical data set whose similarity with the feature vector is greater than a% and less than the similarity threshold is selected to establish an analogy set, and the processing time of the data change information is determined according to the analogy set.

9. The database processing method based on message queue and synchronous processing according to claim 8 is characterized in that: Determining the processing time of the data change information according to the analogy set includes: determining a median of the historical processing durations in the analog set; Extracting the historical processing durations that are greater than the median among all the historical processing durations to construct a first data set; Extracting the historical processing durations that are shorter than the median among all the historical processing durations to construct a second data set; Determining a processing time of the data change information according to the first data set and the second data set; Among them, T represents the processing time of data change information, N represents the number of historical processing times in the first data set, Ti represents the i-th historical processing time in the first data set, T0 represents the median of the historical processing time in the analogy set, M represents the number of historical processing times in the second data set, and Tj represents the j-th historical processing time in the second data set.

10. A database processing system based on message queue and synchronous processing, used for applying the database processing method based on message queue and synchronous processing as claimed in claim 1, characterized in that: include: The collection unit is configured to collect data information of the database master library, and when data changes occur in the database master library, call the API of Active MQ to insert the data change information; A sorting unit is configured to collect the data change information and determine the message queue duration according to the change information, collect all data change information within the message queue duration, and build a message data chain; an analysis unit configured to analyze each data change information in the message data chain using a neural network model, obtain importance data of each data change information, split the message data chain according to the importance data, and combine the message data chain into a new message data chain, wherein the new message data chain includes a priority processing data chain and a non-priority processing data chain; an identification unit configured to extract a feature vector of each of the data change messages in the priority processing data chain, determine a processing time of each of the data change messages according to the feature vector, sort the priority processing data chains according to the processing time, and obtain a final message data chain; The processing unit is configured to add the final message data chain to the List data structure of Redis, and obtain the distributed lock according to the order of the final message data chain to perform message processing.

Images