A database processing method and system based on message queue and synchronous processing

Abstract

The present invention relates to the technical field of data processing, and discloses a database processing method and system based on message queue and synchronous processing. The method includes: when data changes occur in the main database, calling the API of Active MQ to insert data change information; collecting the data change information to determine the message queue duration, and collecting all the data change information to construct a message data chain; obtaining the importance data of each data change information, splitting and combining the message data chain into a new message data chain; extracting the feature vectors of each data change message in the priority processing data chain to determine the processing duration, and obtaining the final message data chain according to the processing duration; adding the final message data chain to the List data structure of Redis, and obtaining a distributed lock in the order of the final message data chain for message processing. This application realizes dynamically adjusting the message processing strategy according to business requirements, improves the efficiency and accuracy of data synchronization, and ensures that data synchronization can be achieved in a high-concurrency environment.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and more particularly, to a database processing method and system based on message queue and synchronous processing. Background Art

[0002] With the rapid development of information technology, enterprises rely on the real-time processing and synchronization of large-scale data in their daily operations. Especially in database systems, the real-time update and synchronization of data have an important impact on aspects such as the operation efficiency, decision-making support, and customer experience of enterprises. Currently, most enterprises adopt a master-slave synchronization architecture for databases, where the master database is responsible for data write operations, and the slave databases are responsible for data reading and backup. In this architecture, data changes occurring in the master database need to be synchronized to each slave database in a timely manner to ensure data consistency and reliability.

[0003] However, in practical applications, master-slave database synchronization faces some problems. For example, traditional database synchronization methods fail to effectively improve real-time performance, resulting in slow processing speeds in high-concurrency scenarios. In the scenario of concurrent processing by multiple consumers, the processing order of the message queue is prone to chaos, leading to data inconsistency or synchronization failure. Especially when the load is high, the coordinated processing among consumers becomes a bottleneck. Most current message processing adopts the first-in-first-out (FIFO) principle and cannot dynamically adjust the priority of messages according to business requirements, resulting in some important data changes not being responded to in a timely manner and affecting the normal operation of the business.

[0004] Therefore, it is necessary to design a database processing method and system based on message queue and synchronous processing to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes a database processing method based on message queue and synchronous processing, aiming to solve the problems in the current master-slave database synchronization technology, such as poor backup effect, low backup speed, and lack of dynamic adjustment of message priority caused by chaotic message queue processing.

[0006] On the one hand, the present invention proposes a database processing method based on message queue and synchronous processing, including:

[0007] Collect data information of the database master. When a data change occurs in the database master, call the API of Active MQ to insert the data change information.

[0008] Collect the data change information and determine the message queue duration according to the change information. Collect all data change information within the message queue duration and construct a message data chain.

[0009] Analyze each data change information in the message data chain using a neural network model to obtain the importance data of each data change information, and split and combine the message data chain into a new message data chain according to the importance data. The new message data chain includes a priority processing data chain and a non-priority processing data chain;

[0010] Extract the feature vectors of each data change message in the priority processing data chain, determine the processing duration of each data change message according to the feature vectors, and sort the priority processing data chain according to the processing duration to obtain the final message data chain;

[0011] Add the final message data chain to the List data structure of Redis, and obtain a distributed lock in the order of the final message data chain for message processing.

[0012] Further, when collecting the data change information and determining the message queue duration according to the change information, it includes:

[0013] Compare the data change amount of the data change message with a first preset change amount and a second preset change amount respectively, and determine the message queue duration according to the comparison result. The first preset change amount is less than the second preset change amount;

[0014] When the data change amount is less than or equal to the first preset change amount, determine that the message queue duration is the first preset duration; when the data change amount is greater than the first preset change amount and less than or equal to the second preset change amount, determine that the message queue duration is the second preset duration; when the data change amount is greater than the second preset change amount, determine that the message queue duration is the third preset duration; the first preset duration is greater than the second preset duration, and the second preset duration is greater than the third preset duration.

[0015] Further, when analyzing each data change information in the message data chain using a neural network model to obtain the importance data of each data change information, it includes:

[0016] Obtain historical data change information, and construct a historical data set according to the historical data change information;

[0017] Sample the historical data set according to a preset ratio to obtain a training subset and a test subset;

[0018] Obtain a pre-selected neural network model, iteratively train the neural network model according to the training subset, evaluate the iteratively trained neural network model according to the test subset, and obtain the final neural network model.

[0019] Further, when analyzing each data change information in the message data chain by using a neural network model to obtain the importance data of each data change information, it further includes:

[0020] Judging whether to stop iterative training according to the evaluation value;

[0021] If the evaluation value of the neural network model after the current iterative training is less than the evaluation value of the neural network model after the previous iterative training, then reduce the amplitude of the change of the neural network model in the gradient direction, and continue iterative training until the preset number of iterations is reached;

[0022] If the evaluation value of the neural network model after the current iterative training is greater than or equal to the evaluation value of the neural network model after the previous iterative training, stop iterative training.

[0023] Further, when splitting and combining the message data chain into a new message data chain according to the importance data, it includes:

[0024] Obtain an initial data chain, where the initial data chain includes a first initial data chain and a second initial data chain. Each initial data chain is provided with a plurality of data chain nodes and connection nodes;

[0025] Generate a chain-changing mark for all data change information on the message data chain that is greater than or equal to the importance threshold;

[0026] Transfer all data change information carrying the chain-changing mark to the first initial data chain to obtain the priority processing data chain;

[0027] Transfer the remaining data change information on the message data chain to the second initial data chain to obtain the non-priority processing data chain.

[0028] Further, when sorting the priority processing data chain according to the processing duration to obtain the final message data chain, it includes:

[0029] Perform Z-score standardization processing on the feature vectors of all data change messages in the priority processing data chain. The feature vectors include the data change amount, the cross-table degree of the data change, and the change frequency;

[0030] Determine the initialization neighborhood radius through a k-distance graph, and determine MinPts to be 6;

[0031] Scan all points, and find the points whose number of points in the neighborhood is greater than or equal to MinPts as core points;

[0032] Starting from each of the core points, check the points in its neighborhood; if the points in the neighborhood are core points, continue to expand the cluster; if the points in the neighborhood are boundary points, add them to the current cluster; if a point is not in the neighborhood of any core point and cannot form a cluster with other points, it is marked as a noise point;

[0033] When the number of core points is less than the number of data change information in the priority processing data chain, it is determined that there are similar change data in the priority processing data chain;

[0034] When the number of core points is equal to the number of data change information in the priority processing data chain, it is determined that there are no similar change data in the priority processing data chain.

[0035] Further, when sorting the priority processing data chain according to the processing duration to obtain the final message data chain, it further includes:

[0036] Determine the processing duration of each data change information in the priority processing data chain;

[0037] When it is determined that there are no similar change data in the priority processing data chain, sort the priority processing data chain from high to low according to the processing duration to obtain the final message data chain;

[0038] When it is determined that there are similar change data in the priority processing data chain, obtain the total processing duration of each similar change data according to the processing duration of each data change information in the similar change data, and sort the priority processing data chain from high to low according to the total processing duration, and from high to low according to the processing duration in the similar change data to obtain the final message data chain.

[0039] Further, when determining the processing duration of each data change information in the priority processing data chain, it includes:

[0040] Compare the feature vector with the historical data set, and determine the processing duration of each data change information according to the comparison result;

[0041] When there is data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, use the historical processing duration of the historical data corresponding to the maximum similarity as the processing duration of the data change information;

[0042] When there is no data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, select the data in the historical data set whose similarity with the feature vector is greater than a% and less than the similarity threshold to establish an analogy set, and determine the processing duration of the data change information according to the analogy set.

[0043] Further, when determining the processing duration of the data change information according to the analogy set, it includes:

[0044] Determine the median of the historical processing durations in the analogy set;

[0045] Extract the historical processing durations greater than the median among all the historical processing durations to construct a first data set;

[0046] Extract the historical processing durations less than the median among all the historical processing durations to construct a second data set;

[0047] Determine the processing duration of the data change information according to the first data set and the second data set;

[0048]

[0049] Wherein, T represents the processing duration of the data change information, N represents the number of historical processing durations in the first data set, Ti represents the i-th historical processing duration in the first data set, T0 represents the median of the historical processing durations in the analogy set, M represents the number of historical processing durations in the second data set, and Tj represents the j-th historical processing duration in the second data set.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows: By collecting the real-time data change information of the main database of the database and inserting it into the message queue through Active MQ, the instant changes of the data are effectively captured. The neural network model is used to analyze the message data chain, dynamically evaluate the priority and importance of each data change, so as to realize the intelligent division of message priorities. By separating the data chains to be processed preferentially from those not to be processed preferentially and optimizing the processing duration and order of each message according to the feature vectors, it is ensured that important data changes can be processed preferentially, improving the processing efficiency and real-time performance of the system. Combining the List data structure of Redis and the distributed lock mechanism, the message order problem in multi-consumer concurrent processing is solved, ensuring the orderliness and consistency of message processing, and avoiding data synchronization failures caused by concurrent conflicts. The performance and stability of the backup system are improved, the message processing strategy can be dynamically adjusted according to business requirements, the efficiency and accuracy of data synchronization are improved, and it is ensured that enterprises can achieve efficient and reliable data synchronization in a high-concurrency environment.

[0051] On the other hand, the present application also provides a database processing system based on message queue and synchronous processing for applying the above-mentioned database processing method based on message queue and synchronous processing, including:

[0052] The acquisition unit is configured to acquire the data information of the main database. When there are data changes in the main database, it calls the API of Active MQ to insert the data change information.

[0053] The sorting unit is configured to collect the data change information and determine the message queue duration according to the change information, collect all the data change information within the message queue duration, and construct a message data chain.

[0054] The analysis unit is configured to analyze each data change information in the message data chain using a neural network model, obtain the importance data of each data change information, and split and combine the message data chain according to the importance data into a new message data chain, where the new message data chain includes a priority processing data chain and a non-priority processing data chain.

[0055] The recognition unit is configured to extract the feature vectors of each data change message in the priority processing data chain, determine the processing duration of each data change message according to the feature vectors, and sort the priority processing data chain according to the processing duration to obtain a final message data chain.

[0056] The processing unit is configured to add the final message data chain to the List data structure of Redis and obtain a distributed lock in the order of the final message data chain for message processing.

[0057] It can be understood that the above database processing method and system based on message queue and synchronous processing have the same beneficial effects, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0059] Figure 1 is a flowchart of the database processing method based on message queue and synchronous processing provided by the embodiment of the present invention;

[0060] Figure 2 is a structural block diagram of the database processing system based on message queue and synchronous processing provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0062] In some embodiments of the present application, referring to Figure 1 as shown, a database processing method based on message queue and synchronous processing includes:

[0063] S100: Collect the data information of the main database of the database. When data changes occur in the main database of the database, call the API of ActiveMQ to insert the data change information.

[0064] S200: Collect the data change information and determine the message queue duration according to the change information. Collect all the data change information within the message queue duration and construct a message data chain.

[0065] S300: Analyze each data change information in the message data chain by using a neural network model to obtain the importance data of each data change information. Split and combine the message data chain according to the importance data to form a new message data chain, and the new message data chain includes a priority processing data chain and a non-priority processing data chain.

[0066] S400: Extract the feature vectors of each data change message in the priority processing data chain, determine the processing duration of each data change message according to the feature vectors, and sort the priority processing data chain according to the processing duration to obtain the final message data chain.

[0067] S500: Add the final message data chain to the List data structure of Redis, and obtain a distributed lock in the order of the final message data chain for message processing.

[0068] Specifically, in S100, by monitoring the data changes in the main database of the database in real time, any insert, update, or delete operations occurring in the database can be captured in a timely manner. When data changes occur in the main database, the API of Active MQ is called to insert the change information into the message queue in the form of a message. Technologies such as database triggers, log listening, or database change capture can be used to monitor the data changes in the main database of the database in real time, and send the change messages to the message queue through the Active MQ API. Active MQ is an open-source message middleware that allows applications to send and receive messages asynchronously through the message queue. The API refers to the set of interfaces and methods used to interact with the message middleware, usually including operations on the client side and the server side, such as sending, receiving, and managing the message queue, etc.

[0069] In S200, the collected data change information is classified, and the duration of the message queue (i.e., the data collection time window) is determined. Within this duration, all relevant data change information will be collected and organized into a message data chain. Ensure that the system can process and manage all relevant change information. By setting the duration of the message queue, ensure that the system obtains all change information within a specific time window, and aggregates this information into a data chain in chronological order. Each change message will queue up in the message queue waiting to be processed. In S300, a neural network model is introduced to analyze each data change information and calculate its importance. Each changed piece of information will be evaluated with an importance value according to the data change location. Based on these importance values, the message data chain is split into a priority processing data chain and a non-priority processing data chain, and the data to be processed first will be arranged for priority consumption. The neural network model can predict the business importance of data changes through training data. This process generates an importance score for each data change based on feature engineering and performs a splitting operation based on the score. In S400, the feature vectors of each data change message in the priority processing data chain are extracted, and these feature vectors are used to evaluate the processing duration of each message, and the messages are sorted according to the processing duration. The sorted data chain will be processed in the order of priority. The feature vector is a numerical description of each message, containing factors that affect the processing duration, such as the amount of data change, the degree of cross-table data change, and the change frequency, etc. In S500, after all messages are sorted and ready, the final message data chain will be added to the List data structure of Redis to ensure processing in order. Each message will be consumed only after obtaining a distributed lock, ensuring that only one consumer can process a specific message, thus avoiding concurrent conflicts and disorder in the order. Redis's List data structure is used to store the message queue arranged in order. The distributed lock (such as SETNX or Redlock based on Redis) ensures that when multiple consumers process messages simultaneously, there will be no situation of duplicate message consumption or disorder in the consumption order.

[0070] It is understandable that by collecting the real-time data changes of the main database of the database and inserting them into the message queue, the real-time nature of data synchronization is ensured. By analyzing and evaluating the priority of data changes through a neural network model, the message data chain is intelligently divided and sorted to ensure that important business data can be processed first, avoiding the delay in processing important data caused by the traditional FIFO method. Using the Redis List data structure and distributed lock technology, the message order and consistency during concurrent processing by multiple consumers are ensured, effectively avoiding concurrent conflicts and data inconsistency problems. It can maintain efficient message processing in a high-concurrency environment, improving the real-time nature, accuracy, and reliability of data synchronization, and adapting to data processing and synchronization scenarios under complex business requirements.

[0071] In some embodiments of the present application, when collecting data change information and determining the message queue duration according to the change information, it includes: comparing the data change amount of the data change message with a first preset change amount and a second preset change amount respectively, and determining the message queue duration according to the comparison result, where the first preset change amount is less than the second preset change amount.

[0072] Specifically, when the data change amount is less than or equal to the first preset change amount, the message queue duration is determined to be the first preset duration. When the data change amount is greater than the first preset change amount and less than or equal to the second preset change amount, the message queue duration is determined to be the second preset duration. When the data change amount is greater than the second preset change amount, the message queue duration is determined to be the third preset duration. The first preset duration is greater than the second preset duration, and the second preset duration is greater than the third preset duration.

[0073] It is understandable that by flexibly adjusting the message queue duration according to the actual data change amount instead of using a fixed duration, the problem of wasting resources in the case of low change amounts or causing processing timeouts in the case of high change amounts is avoided. By adjusting the queue duration according to the change amount, it can be quickly processed in the case of small-scale changes, while sufficient time is allocated for processing in the case of large-scale changes, avoiding processing delays caused by insufficient queue duration. Larger data changes usually require more time and resources for processing. Through reasonable queue duration allocation, large-scale data changes can be effectively handled, reducing the risk of concurrent conflicts and improving the overall processing efficiency.

[0074] In some embodiments of the present application, when using a neural network model to analyze each data change information in the message data chain to obtain the importance data of each data change information, it includes:

[0075] Obtaining historical data change information and constructing a historical data set according to the historical data change information.

[0076] The historical data set is sampled according to a preset ratio to obtain a training subset and a test subset.

[0077] An artificially selected neural network model is obtained, and the neural network model is iteratively trained according to the training subset, and the iteratively trained neural network model is evaluated according to the test subset to obtain a final neural network model.

[0078] In some embodiments of the present application, when using a neural network model to analyze each data change information in the message data chain and obtain the importance data of each data change information, it further includes:

[0079] Determine whether to stop the iterative training according to the evaluation value.

[0080] If the evaluation value of the neural network model after the current iterative training is less than the evaluation value of the neural network model after the previous iterative training, the amplitude of the change of the neural network model in the gradient direction is reduced, and the iterative training is continued until the preset number of iterations is reached.

[0081] If the evaluation value of the neural network model after the current iterative training is greater than or equal to the evaluation value of the neural network model after the previous iterative training, the iterative training is stopped.

[0082] It is understandable that a large amount of historical data change information is required for training a neural network model. This historical data change information includes insert, update, and delete operations in the database, and information such as the time, type, content of the change, and the tables and fields involved. By collecting and organizing this historical change data, a historical dataset is constructed, which will serve as the training basis for the neural network. The historical data change information is collected through database logs. The data comes from the change records, triggers of the database, or the logs of the application. The collected historical data is cleaned and organized, and the useful information (such as the change time, operation type, changed tables and fields) is converted into a format that can be input into the neural network. The dataset is divided into a training subset and a test subset. The training subset is used for training the model, and the test subset is used to verify the accuracy of the model to ensure that the model does not overfit. According to a preset ratio (for example, 80% for training and 20% for testing), the training subset and the test subset are sampled from the historical dataset. Select a neural network model (such as a multi-layer perceptron or a convolutional neural network), and use the training subset to iteratively train the neural network model. The prediction error is minimized by gradually adjusting the parameters of the model. The training process will continue until the evaluation value of the model meets the preset stop condition. The training subset is used for forward propagation and backward propagation of the neural network, and the weights of the network are gradually adjusted through an optimization algorithm (such as the gradient descent method). The trained model is evaluated using the test subset, and the effect of the model is measured through evaluation metrics (such as accuracy, precision, recall, etc.). The evaluation value of the model will reflect the current prediction ability of the model. When the evaluation value continues to improve, it indicates that the model is gradually improving. If it is found that the evaluation value of the model after the current iterative training is lower than the previous one, it means that the model has overfitted or been over-trained. At this time, the learning rate needs to be adjusted or the training method needs to be changed. If the evaluation value of the current model is less than or equal to the previous evaluation value, it means that the model has stabilized and the training stops. Otherwise, continue training according to the adjustment strategy until the preset maximum number of iterations or the loss value converges. After the model training is completed, the neural network analyzes each data change information and outputs an importance data. This importance data is a scalar value, indicating the relative importance of this data change information in database synchronization. The calculation of this importance data will be based on the location of the data change. For example, when the change information occurs in a key table or an important field of the main database, a higher priority will be assigned.

[0083] It is understandable that the historical information of data changes is analyzed by a neural network model to automatically identify which data changes are more critical and urgent. Considering the location of data in the database, it ensures that the core business data can be processed preferentially. The neural network model automatically learns and optimizes the priority evaluation based on historical data. It avoids the problem that manual rules cannot cover all scenarios. Through the adaptive training and adjustment of the neural network, it can automatically adjust the processing strategy according to the changes in business requirements and database environment, improving the degree of automation and adaptability.

[0084] In some embodiments of the present application, when splitting and combining the message data chain into a new message data chain according to the importance data, it includes: obtaining an initial data chain, where the initial data chain includes a first initial data chain and a second initial data chain. Each initial data chain is provided with multiple data chain nodes and connection nodes. Generating a chain-changing mark for all data change information on the message data chain that is greater than or equal to the importance threshold.

[0085] Specifically, transfer all data change information carrying the chain-changing mark to the first initial data chain to obtain a preferentially processed data chain. Transfer the remaining data change information on the message data chain to the second initial data chain to obtain a non-preferentially processed data chain.

[0086] In some embodiments of the present application, when sorting the preferentially processed data chain according to the processing duration to obtain the final message data chain, it includes: performing Z-score standardization processing on the feature vectors of all data change messages in the preferentially processed data chain. The feature vectors include the amount of data change, the degree of cross-table data change, and the change frequency.

[0087] Determine the initial neighborhood radius through the k-distance graph, and determine MinPts as 6.

[0088] Scan all points to find the points whose number of points in the neighborhood is greater than or equal to MinPts as core points.

[0089] Starting from each core point, check the points in its neighborhood. If the points in the neighborhood are core points, continue to expand the cluster. If the points in the neighborhood are boundary points, add them to the current cluster. If a point is not in the neighborhood of any core point and cannot form a cluster with other points, it is marked as a noise point.

[0090] Specifically, when the number of core points is less than the number of data change information in the preferentially processed data chain, it is determined that there are similar change data in the preferentially processed data chain. When the number of core points is equal to the number of data change information in the preferentially processed data chain, it is determined that there is no similar change data in the preferentially processed data chain.

[0091] In some embodiments of the present application, when sorting the priority processing data chain according to the processing duration to obtain the final message data chain, it further includes: determining the processing duration of each data change information in the priority processing data chain. When it is determined that there is no same-type change data in the priority processing data chain, the priority processing data chain is sorted from high to low according to the processing duration to obtain the final message data chain. When it is determined that there is same-type change data in the priority processing data chain, the total processing duration of each same-type change data is obtained according to the processing duration of each data change information in the same-type change data, and the priority processing data chain is sorted from high to low according to the total processing duration and from high to low according to the processing duration in the same-type change data to obtain the final message data chain.

[0092] It can be understood that in the priority processing data chain, the processing duration of the data change information serves as the sorting basis. To better sort and optimize these information, these information are standardized and evaluated according to feature vectors such as the data change amount, the degree of data change across tables, and the change frequency. The feature vectors of each data change information in the priority processing data chain are Z-score standardized, so that the data change information with different features can be compared on the same scale. Through the standardized feature vectors, the information in the priority processing data chain will be sorted according to the processing duration to ensure that the most important or most urgent data changes are processed first. Through the k-distance graph, the MinPts clustering algorithm is used to analyze the data change information in the priority processing data chain. Identify whether there is same-type change data in the data in the priority processing data chain, that is, whether there is a group of similar data change information that needs to be processed together. Further optimize the processing order of data changes and reduce redundant processing. By calculating the neighborhood radius of the data change information, it is determined whether each data change information is a core point. When there is same-type change data in the priority processing data chain, in addition to considering the processing duration of each data change information, it is also necessary to calculate the total processing duration of the same-type change data. If the total duration of the same-type change data is long, these data are processed first, and the processing duration of each data therein is arranged in descending order to achieve optimal processing.

[0093] It can be understood that by splitting the message data chain into a priority processing data chain and a non-priority processing data chain according to the importance data and the chain change mark, it is possible to ensure the priority processing of important data changes and improve the response speed and accuracy of the synchronization of business core data. Through the sorting and processing mechanism, it is possible to effectively reduce the delay problems caused by too long data processing duration or frequent data changes, and improve the overall efficiency and real-time performance of the synchronization system. By combining importance data, processing duration, and same-type data clustering analysis, the entire synchronization system is more adaptable to complex database environments and high-concurrency scenarios, and can flexibly adjust the processing strategy according to actual needs.

[0094] In some embodiments of the present application, when determining the processing duration of each data change information in the priority processing data chain, it includes: comparing the feature vector with the historical data set, and determining the processing duration of each data change information according to the comparison result.

[0095] Specifically, when there is data in the historical data set with a similarity greater than the similarity threshold to the feature vector, the historical processing duration of the historical data corresponding to the maximum similarity is used as the processing duration of the data change information. When there is no data in the historical data set with a similarity greater than the similarity threshold to the feature vector, data with a similarity greater than a% and less than the similarity threshold to the feature vector in the historical data set is selected to establish an analogy set, and the processing duration of the data change information is determined according to the analogy set.

[0096] In some embodiments of the present application, when determining the processing duration of the data change information according to the analogy set, it includes: determining the median of the historical processing durations in the analogy set. Extracting the historical processing durations greater than the median among all the historical processing durations to construct a first data set. Extracting the historical processing durations less than the median among all the historical processing durations to construct a second data set. Determining the processing duration of the data change information according to the first data set and the second data set.

[0097]

[0098] Wherein, T represents the processing duration of the data change information, N represents the number of historical processing durations in the first data set, Ti represents the i-th historical processing duration in the first data set, T0 represents the median of the historical processing durations in the analogy set, M represents the number of historical processing durations in the second data set, and Tj represents the j-th historical processing duration in the second data set.

[0099] It can be understood that by comparing with historical data, the processing duration of each data change is predicted, avoiding system delay or overprocessing caused by overly simple prediction. Through the dynamic adjustment of the similarity threshold and the analogy set, the prediction of the processing duration can be flexibly adjusted according to the actual data change situation, making it more in line with the actual business requirements. Through the median analysis of the historical data set, processing resources can be efficiently allocated to ensure the timely processing of important data and prevent the progress from being slowed down by a large number of small change information. By comparing the feature vector with the historical data, the error caused by overly broad processing duration estimation is reduced, avoiding unnecessary processing delay or resource waste.

[0100] In the above embodiments, by collecting the real-time data change information of the database master and inserting it into the message queue through Active MQ, the instant changes of the data are effectively captured. The neural network model is used to analyze the message data chain, dynamically evaluate the priority and importance of each data change, so as to realize the intelligent division of message priorities. By separating the data chains to be processed preferentially from those not to be processed preferentially and optimizing the processing duration and order of each message according to the feature vectors, it is ensured that important data changes can be processed preferentially, improving the processing efficiency and real-time performance of the system. Combining the List data structure of Redis and the distributed lock mechanism solves the problem of message order in concurrent processing by multiple consumers, ensuring the order and consistency of message processing and avoiding data synchronization failures caused by concurrent conflicts. It improves the performance and stability of the backup system, realizes the dynamic adjustment of message processing strategies according to business requirements, improves the efficiency and accuracy of data synchronization, and ensures that enterprises can achieve efficient and reliable data synchronization in a high-concurrency environment.

[0101] In another preferred manner based on the above embodiments, refer to Figure 2 As shown, this embodiment provides a database processing system based on message queue and synchronous processing, which is applied to the above database processing method based on message queue and synchronous processing, and includes:

[0102] A collection unit, configured to collect the data information of the database master, and when a data change occurs in the database master, call the API of Active MQ to insert the data change information;

[0103] An arrangement unit, configured to collect the data change information and determine the message queue duration according to the change information, collect all the data change information within the message queue duration, and construct a message data chain;

[0104] An analysis unit, configured to use a neural network model to analyze each data change information in the message data chain, obtain the importance data of each data change information, and split and combine the message data chain according to the importance data into a new message data chain, where the new message data chain includes a preferentially processed data chain and a non-preferentially processed data chain;

[0105] An identification unit, configured to extract the feature vectors of each data change message in the preferentially processed data chain, determine the processing duration of each data change message according to the feature vectors, and sort the preferentially processed data chain according to the processing duration to obtain a final message data chain;

[0106] A processing unit, configured to add the final message data chain to the List data structure of Redis, and obtain a distributed lock in the order of the final message data chain for message processing.

[0107] It can be understood that by collecting the real-time data change information of the main database and inserting it into the message queue through ActiveMQ, the instant changes of the data can be effectively captured. The neural network model is used to analyze the message data chain, dynamically evaluate the priority and importance of each data change, so as to realize the intelligent message priority division. By separating the data chains to be processed preferentially from those not to be processed preferentially and optimizing the processing duration and order of each message according to the feature vectors, it is ensured that important data changes can be processed preferentially, improving the system processing efficiency and real-time performance. Combining the List data structure of Redis and the distributed lock mechanism solves the problem of message order in concurrent processing by multiple consumers, ensuring the sequentiality and consistency of message processing and avoiding data synchronization failures caused by concurrent conflicts. It improves the performance and stability of the backup system, realizes the dynamic adjustment of the message processing strategy according to business requirements, improves the efficiency and accuracy of data synchronization, and ensures that enterprises can achieve efficient and reliable data synchronization in a high-concurrency environment.

[0108] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 the functions specified in one block or multiple blocks.

[0110] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 the functions specified in one block or multiple blocks.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or one block or a plurality of blocks. Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps for implementing the functions specified in one block or a plurality of blocks.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent replacements can still be made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A database processing method based on message queue and synchronous processing, characterized in that, Including: Collect the data information of the main database of the collection database. When there is a data change in the main database of the said database, call the API of Active MQ to insert the data change information; Collect the said data change information and determine the message queue duration according to the change information. Collect all the data change information within the message queue duration and construct a message data chain; Use a neural network model to analyze each data change information in the message data chain, obtain the importance data of each data change information, and split and combine the message data chain according to the importance data into a new message data chain. The new message data chain includes a priority processing data chain and a non-priority processing data chain; Extract the feature vectors of each data change message in the priority processing data chain, determine the processing duration of each data change message according to the feature vectors, and sort the priority processing data chain according to the processing duration to obtain the final message data chain; Add the final message data chain to the List data structure of Redis, and obtain a distributed lock in the order of the final message data chain for message processing; When sorting the priority processing data chain according to the processing duration to obtain the final message data chain, it also includes: Determine the processing duration of each data change information in the priority processing data chain; When it is determined that there is no similar change data in the priority processing data chain, sort the priority processing data chain from high to low according to the processing duration to obtain the final message data chain; When it is determined that there is similar change data in the priority processing data chain, obtain the total processing duration of each similar change data according to the processing duration of each data change information in the similar change data, and sort the priority processing data chain from high to low according to the total processing duration, and from high to low according to the processing duration in the similar change data to obtain the final message data chain; When determining the processing duration of each data change information in the priority processing data chain, it includes: Compare the feature vectors with the historical data set, and determine the processing duration of each data change information according to the comparison result; When there is data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, use the historical processing duration of the historical data corresponding to the maximum similarity as the processing duration of the data change information; When there is no data in the historical data set whose similarity with the feature vector is greater than the similarity threshold, select the data in the historical data set whose similarity with the feature vector is greater than a% and less than the similarity threshold to establish an analogy set, and determine the processing duration of the data change information according to the analogy set; When determining the processing duration of the data change information according to the analogy set, it includes: Determine the median of the historical processing durations in the analogy set; Extract the historical processing durations greater than the median among all the historical processing durations to construct a first data set; Extract the historical processing durations less than the median among all the historical processing durations to construct a second data set; Determine the processing duration of the data change information according to the first data set and the second data set; Among them, T represents the processing duration of data change information, N represents the number of historical processing durations in the first dataset, Ti represents the i-th historical processing duration in the first dataset, T0 represents the median of the historical processing durations in the analogy set, M represents the number of historical processing durations in the second dataset, and Tj represents the j-th historical processing duration in the second dataset.

2. The database processing method based on message queue and synchronous processing according to claim 1, characterized in that When collecting the data change information and determining the message queue duration according to the change information, it includes: Compare the data change amount of the data change message with a first preset change amount and a second preset change amount respectively, and determine the message queue duration according to the comparison result, where the first preset change amount is less than the second preset change amount; When the data change amount is less than or equal to the first preset change amount, determine that the message queue duration is the first preset duration; when the data change amount is greater than the first preset change amount and less than or equal to the second preset change amount, determine that the message queue duration is the second preset duration; when the data change amount is greater than the second preset change amount, determine that the message queue duration is the third preset duration; the first preset duration is greater than the second preset duration, and the second preset duration is greater than the third preset duration.

3. The database processing method based on message queue and synchronous processing according to claim 1, characterized in that When using a neural network model to analyze each data change information in the message data chain and obtain the importance data of each data change information, it includes: Obtain historical data change information and construct a historical data set according to the historical data change information; Sample the historical data set according to a preset ratio to obtain a training subset and a test subset; Obtain a pre-selected neural network model, perform iterative training on the neural network model according to the training subset, evaluate the iteratively trained neural network model according to the test subset, and obtain the final neural network model.

4. The database processing method based on message queue and synchronous processing according to claim 3, wherein When using a neural network model to analyze each data change information in the message data chain and obtain the importance data of each data change information, it also includes: Judge whether to stop iterative training according to the evaluation value; If the evaluation value of the neural network model after the current iterative training is less than the evaluation value of the neural network model after the previous iterative training, reduce the amplitude of the change of the neural network model in the gradient direction and continue iterative training until the preset number of iterations is reached; If the evaluation value of the neural network model after the current iterative training is greater than or equal to the evaluation value of the neural network model after the previous iterative training, stop iterative training.

5. The database processing method based on message queue and synchronous processing according to claim 3, characterized in that, When splitting and combining the message data chain into a new message data chain according to the importance data, it includes: Obtain an initial data chain, where the initial data chain includes a first initial data chain and a second initial data chain, and multiple data chain nodes and connection nodes are arranged on each initial data chain; Generate a chain change mark for all data change information on the message data chain that is greater than or equal to the importance threshold; Transfer all data change information with a chain change mark to the first initial data chain to obtain the priority processing data chain; Transfer the remaining data change information on the message data chain to the second initial data chain to obtain the non-priority processing data chain.

6. The database processing method based on message queue and synchronous processing according to claim 1, wherein When sorting the priority processing data chain according to the processing duration to obtain the final message data chain, it includes: Perform Z-score normalization on the feature vectors of all data change messages in the priority processing data chain, where the feature vectors include the amount of data change, the degree of cross-table data change, and the change frequency; Determine the initial neighborhood radius through the k-distance graph and set MinPts to 6; Scan all points and identify the points with the number of points in the neighborhood greater than or equal to MinPts as core points; Starting from each core point, check the points in its neighborhood. If the point in the neighborhood is a core point, continue to expand the cluster. If the point in the neighborhood is a border point, add it to the current cluster. If a point is not in the neighborhood of any core point and cannot form a cluster with other points, it is marked as a noise point; When the number of core points is less than the number of data change information in the priority processing data chain, it is determined that there are similar change data in the priority processing data chain; When the number of core points is equal to the number of data change information in the priority processing data chain, it is determined that there is no similar change data in the priority processing data chain.

7. A database processing system based on message queue and synchronous processing, which is used to apply the database processing method based on message queue and synchronous processing as described in claim 1, and is characterized in that, It includes: An acquisition unit configured to acquire the data information of the main database of the database. When data changes occur in the main database of the database, it calls the API of Active MQ to insert data change information; An arrangement unit configured to acquire the data change information and determine the message queue duration according to the change information, acquire all data change information within the message queue duration, and construct a message data chain; An analysis unit configured to analyze each data change information in the message data chain using a neural network model to obtain the importance data of each data change information, and split and combine the message data chain according to the importance data into a new message data chain, where the new message data chain includes a priority processing data chain and a non-priority processing data chain; An identification unit configured to extract the feature vectors of each data change message in the priority processing data chain, determine the processing duration of each data change message according to the feature vectors, sort the priority processing data chain according to the processing duration, and obtain the final message data chain; A processing unit configured to add the final message data chain to the List data structure of Redis and obtain a distributed lock in the order of the final message data chain for message processing.

Images