Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method which can be applied to the technical field of cloud computing and the field of financial science and technology. The data processing method comprises the following steps: analyzing an obtained data table to determine a data synchronization strategy corresponding to the data table; data extraction is conducted on the data table through a data synchronization strategy, data to be synchronized are obtained, and the data synchronization strategy comprises a data extraction rule associated with the data service logic; the to-be-synchronized data is written into a data synchronization queue of message middleware, and the message middleware is used for synchronizing the data in the data synchronization queue to the target node; and when it is determined that the data received by the target node is inconsistent with the to-be-synchronized data, repairing the synchronized data according to a fault repairing rule corresponding to the data service logic.

Description

Technical Field

[0001] The present disclosure relates to the fields of cloud computing technology and financial technology, and in particular to data synchronization of a database, and more particularly to a data processing method, device, electronic device, storage medium, and program product. Background Art

[0002] With the rapid development of cloud computing technology, distributed databases have become the core infrastructure supporting many large-scale applications. In a distributed environment, how to ensure the consistency and synchronization of data between various database nodes has become an urgent problem to be solved. In related technologies, data synchronization of data databases is achieved through database synchronization plug-ins.

[0003] In the process of realizing the concept of the present disclosure, there are at least the following problems in the related technology: when the database synchronization plug-in realizes data synchronization, as the amount of data in the distributed database increases, the business logic and conversion rules between data gradually become complicated, which increases the probability of data delay and data loss in the data synchronization process, seriously affecting the synchronization efficiency of data in the distributed database. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data processing method, apparatus, electronic device, storage medium and program product.

[0005] According to the first aspect of the present disclosure, a data processing method is provided, comprising: analyzing an acquired data table to determine a data synchronization strategy corresponding to the above data table; extracting data from the above data table using the above data synchronization strategy to obtain data to be synchronized, wherein the above data synchronization strategy includes data extraction rules associated with data business logic; writing the above data to be synchronized into a data synchronization queue of a message middleware, wherein the above message middleware is used to synchronize the data in the above data synchronization queue to a target node; and when it is determined that the data received by the above target node is inconsistent with the above data to be synchronized, repairing the synchronized data according to the fault repair rules corresponding to the above data business logic.

[0006] According to an embodiment of the present disclosure, the above-mentioned data synchronization strategy is used to extract data from the above-mentioned data table to obtain the data to be synchronized, including: determining a type of data and a type of data in the above-mentioned data table according to the association relationship between the data in the above-mentioned data business logic, wherein the above-mentioned type of data is business-necessary data, and the above-mentioned type of data can be inferred based on the above-mentioned type of data; extracting the above-mentioned type of data and the above-mentioned type of data to obtain intermediate data; verifying the above-mentioned intermediate data according to the data verification rules in the above-mentioned data synchronization strategy to obtain an intermediate verification result; when the above-mentioned intermediate verification result indicates that the above-mentioned type of data has passed the verification, using the data conversion rules in the above-mentioned data synchronization strategy to convert the format of the data that has passed the verification to obtain the above-mentioned data to be synchronized.

[0007] According to an embodiment of the present disclosure, the above method also includes: synchronizing the data in the above data synchronization queue based on the table dimension using the above message middleware; when it is determined that the above target node has completed receiving the above data to be synchronized, verifying the received data based on the above data verification rules to obtain a synchronization verification result; when the above synchronization verification result indicates that the verification has passed, performing a consistency check on the received data and the above data to be synchronized, wherein the above consistency check operation includes a data synchronization power check and a data business logic check.

[0008] According to an embodiment of the present disclosure, the above method also includes: monitoring each data synchronization queue in the above message middleware to obtain queue attribute information, wherein the above queue attribute information includes queue availability status, message accumulation number and message processing efficiency; parsing the configured synchronization verification rules to obtain executable parameters; running the above executable parameters to verify the above queue attribute information; when it is determined that there is an abnormality in the above queue attribute information, logically isolating the data synchronization queue where the abnormality occurs to reject new data writing.

[0009] According to an embodiment of the present disclosure, the synchronized data is repaired according to the fault repair rules corresponding to the above-mentioned data business logic, including: when it is determined that the above-mentioned two types of data are missing in the received data, based on the above-mentioned data business logic, the missing two types of data are supplemented according to the above-mentioned one type of data to obtain supplemented data; determining the available status of the data synchronization queue where the supplemented data is located; when it is determined that the above-mentioned data synchronization queue is unavailable, allocating the above-mentioned supplemented data to the remaining data synchronization queues associated with the above-mentioned data business logic to complete the repair of the missing two types of data.

[0010] According to an embodiment of the present disclosure, the above method also includes: when it is determined that the above type of data is missing in the received data, the data received by the above target node is discarded, and the data synchronization status of the above target node is updated to synchronization failure; when it is determined that the data received by the above target node is consistent with the above data to be synchronized, the above data to be synchronized is registered and updated in the above target node.

[0011] According to an embodiment of the present disclosure, the above method also includes: in response to a received update request for the above data synchronization policy, performing security maintenance on the new data synchronization policy in the above update request; publishing and implementing the above new data synchronization policy that has been maintained to replace the original data synchronization policy.

[0012] The second aspect of the present disclosure provides a data processing device, which includes: a policy determination module, which is used to analyze the acquired data table to determine the data synchronization policy corresponding to the data table; a data extraction module, which is used to extract data from the data table using the data synchronization policy to obtain data to be synchronized, wherein the data synchronization policy includes data extraction rules associated with the data business logic; a data writing module, which is used to write the data to be synchronized into the data synchronization queue of the message middleware, wherein the message middleware is used to synchronize the data in the data synchronization queue to the target node; a data repair module, which is used to repair the synchronized data according to the fault repair rules corresponding to the data business logic when it is determined that the data received by the target node is inconsistent with the data to be synchronized.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] According to the embodiments of the present disclosure, a data synchronization strategy is configured for a data table in advance, and the acquired data table is analyzed to determine the data synchronization strategy corresponding to the data table. The data extraction rules associated with the data business logic in the data synchronization strategy are used to extract data from the data table to obtain the data to be synchronized, so as to facilitate the synchronization of data in different databases, so as to be compatible with multiple synchronization modes and reduce synchronization errors caused by the lack of support for the data synchronization plug-in. In the case where the data received by the target node does not meet the consistency with the data to be synchronized, the synchronized data is repaired based on the fault repair rules corresponding to the data business logic, so as to reduce the frequency of re-synchronization of data, and improve the synchronization efficiency of data in distributed databases by reducing the error rate of data synchronization and the number of restarts of data synchronization operations. . BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario diagram of the data processing method according to the embodiment of the present disclosure is schematically shown;

[0019] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0020] Figure 3 An interaction diagram schematically illustrates a data processing method according to an embodiment of the present disclosure;

[0021] Figure 4 Schematically shows a flowchart of repairing synchronization data according to an embodiment of the present disclosure;

[0022] Figure 5 The structure block diagram of the data processing device according to the embodiment of the present disclosure is schematically shown;

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0030] In order to improve the timeliness and quality of services as much as possible, it is necessary to improve the reliability, timeliness and accuracy of data synchronization between multiple nodes in a distributed database. The existing method of implementing data synchronization through data synchronization plug-ins is difficult to meet the needs of various existing business data. Some database synchronization plug-ins only consider basic synchronization requirements based on data synchronization performance considerations and do not provide flexible policy configuration functions, resulting in the inability to provide more complex logic to handle data consistency and conflict resolution of multiple target database nodes when implementing a 1-to-N synchronization mode.

[0031] For example, some database synchronization plug-ins will cancel foreign key constraints by default when synchronizing to the target database. Therefore, operations such as cascading and deletion of the source database will not be synchronized to the target database, resulting in incomplete data synchronization, which in turn brings certain challenges to the data consistency of the distributed database.

[0032] In addition, idempotent synchronization and logical consistency synchronization require the database system to be able to handle more complex data relationships and states, which may be beyond the design scope of some synchronization plug-ins.

[0033] Based on the above background, the embodiments of the present disclosure provide a data processing method to realize data synchronization, data consistency detection, fault monitoring, and automatic recovery services for multiple database nodes under a distributed architecture. Specifically, the data processing method includes: analyzing the acquired data table to determine the data synchronization strategy corresponding to the data table; extracting data from the data table using the data synchronization strategy to obtain the data to be synchronized, wherein the data synchronization strategy includes data extraction rules associated with the data business logic; writing the data to be synchronized into the data synchronization queue of the message middleware, wherein the message middleware is used to synchronize the data in the data synchronization queue to the target node; when it is determined that the data received by the target node is inconsistent with the data to be synchronized, repairing the synchronized data according to the fault repair rules corresponding to the data business logic.

[0034] Figure 1 The application scenario diagram of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0035] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0036] The staff can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as web browser applications, search applications, instant messaging tools, email clients, etc. (only examples).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The data processing method of the disclosed embodiment is described in detail.

[0042] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0043] like Figure 2As shown, the data processing method of this embodiment includes operations S210 to S240.

[0044] In operation S210 , the acquired data table is analyzed to determine a data synchronization strategy corresponding to the data table.

[0045] In operation S220, data is extracted from the data table using a data synchronization strategy to obtain data to be synchronized, wherein the data synchronization strategy includes data extraction rules associated with the data business logic.

[0046] In operation S230, the data to be synchronized is written into a data synchronization queue of the message middleware, wherein the message middleware is used to synchronize the data in the data synchronization queue to the target node.

[0047] In operation S240, when it is determined that the data received by the target node is inconsistent with the data to be synchronized, the synchronized data is repaired according to a fault repair rule corresponding to the data service logic.

[0048] According to an embodiment of the present disclosure, a data table is a data table that needs to be synchronized, and each data table has a corresponding record for identifying whether the data table is a data table that needs to be synchronized or a data table that does not need data synchronization. The data table to be synchronized can be obtained through an application program interface.

[0049] According to an embodiment of the present disclosure, a data synchronization strategy may include information such as source database node, target database node, source table name, source table field name, target table name, target table field name, synchronization mode, synchronization timeliness, fault repair strategy, data verification rule, idempotence check rule, data conversion rule, synchronization restriction rule, etc. The synchronization mode may include full synchronization and incremental synchronization. The synchronization timeliness may include real-time synchronization, end-of-day synchronization, and scheduled synchronization.

[0050] According to the embodiments of the present disclosure, the acquired data table is analyzed to determine the data synchronization strategy corresponding to the data table. Specifically, the metadata of the data table may be analyzed, including the field (or column) data type, primary key, foreign key, index, etc. in the data table. According to the characteristics of the analyzed data table, a data synchronization strategy corresponding to the data table is determined from multiple data synchronization strategies.

[0051] For example, after analyzing the data table orders, we found that it contains fields such as order_id (primary key), customer_id (foreign key associated with the customers table), and order_date (index). Based on these characteristics, we choose the full synchronization strategy and execute it every night.

[0052] According to an embodiment of the present disclosure, data business logic characterizes the business relationship between different types of data in the same data table. For example, the data table includes class A data and class B data. During the processing of business A, it is necessary to call class A data and then call class B data. In this case, "calling class A data and then calling class B data" is the household business logic between class A data and class B data.

[0053] According to an embodiment of the present disclosure, a data extraction rule represents a rule for extracting data configured for a data table based on data business logic, and the data extraction rule may include information such as data conversion rules and synchronization restriction rules.

[0054] Based on the data extraction rules associated with the data business logic included in the data synchronization strategy corresponding to the data table, data is extracted from the data table to obtain the data to be synchronized.

[0055] According to an embodiment of the present disclosure, the data to be synchronized is placed in the data synchronization queue corresponding to the message middleware, and the record information to be synchronized can be synchronously registered according to the data table dimension for data verification and data migration in the event of a failure. The target node is the target database node, and the message middleware is used to synchronize the data in the data synchronization queue to the target node.

[0056] According to an embodiment of the present disclosure, the consistency check is performed on the data received by the target node and the data to be synchronized according to the data synchronization strategy. If the data received by the target node is inconsistent with the data to be synchronized, the synchronized data is repaired according to the fault repair rules corresponding to the data business logic of the data table in the data synchronization strategy. For example, the data synchronization process is restarted.

[0057] According to the embodiments of the present disclosure, a data synchronization strategy is configured for a data table in advance, and the acquired data table is analyzed to determine the data synchronization strategy corresponding to the data table. The data extraction rules associated with the data business logic in the data synchronization strategy are used to extract data from the data table to obtain the data to be synchronized, so as to facilitate the synchronization of data in different databases, so as to be compatible with multiple synchronization modes and reduce synchronization errors caused by the lack of support for the data synchronization plug-in. In the case where the data received by the target node does not meet the consistency with the data to be synchronized, the synchronized data is repaired based on the fault repair rules corresponding to the data business logic, so as to reduce the frequency of re-synchronization of data, and improve the synchronization efficiency of data in distributed databases by reducing the error rate of data synchronization and the number of restarts of data synchronization operations. .

[0058] According to an embodiment of the present disclosure, a data synchronization strategy is used to extract data from a data table to obtain data to be synchronized, including: determining a type of data and a type of data in the data table according to the association relationship between the data in the data business logic, wherein the type of data is necessary business data, and the type of data can be inferred based on the type of data; extracting the type of data and the type of data to obtain intermediate data; verifying the intermediate data according to the data verification rules in the data synchronization strategy to obtain an intermediate verification result; when the intermediate verification result indicates that the type of data has passed the verification, using the data conversion rules in the data synchronization strategy to convert the format of the data that has passed the verification to obtain the data to be synchronized.

[0059] According to the embodiments of the present disclosure, the first category of data is business essential data. During the data synchronization process, the first category of data can only be obtained by extracting data from the data table; the second category of data is business non-essential data. When the first category of data is determined, the first category of data can be inferred based on the relationship between the data in the data business logic to obtain the second category of data.

[0060] According to an embodiment of the present disclosure, based on the association relationship between data in the data business logic, Class I data and Class II data are determined in the data table. For example, in the data business logic, data that has no upstream dependencies can be regarded as Class I data, and other data can be regarded as Class II data.

[0061] According to an embodiment of the present disclosure, a first type of data and a second type of data are extracted from a data table to obtain intermediate data, wherein the intermediate data represents the data to be synchronized. Data verification rules may include single field verification, uniqueness verification, etc., wherein single field verification includes not empty, comparison operation, inclusion, non-inclusion, etc., and uniqueness verification is used to determine whether there are duplicates and data in the intermediate data. The intermediate data is verified according to the data verification rules in the data synchronization strategy to obtain the intermediate verification result.

[0062] According to an embodiment of the present disclosure, the intermediate verification result includes a verification result for a type of data and a verification result for a type of data. When the intermediate verification result indicates that the type of data has passed the verification and there are data conversion rules in the data synchronization strategy, the format of the type of data and the type of data that have passed the verification are converted based on the data conversion rules to obtain the data to be synchronized.

[0063] According to an embodiment of the present disclosure, data conversion rules include data format conversion, character encoding conversion, etc. Data format conversion is used to convert the data format of the source database node to the data format required by the target node, for example, converting the date format from one representation to another. Character encoding conversion is used to change the character encoding rules to adapt to the system environment of the target node.

[0064] According to the embodiments of the present disclosure, according to the association relationship between data in the data business logic, the data is divided into Class I and Class II, and the intermediate data is verified according to the data verification rules in the data synchronization strategy to obtain the intermediate verification results. When Class I data passes the verification, the data format can be converted, which reduces the requirements for data integrity in the data synchronization process, thereby reducing the difficulty of data synchronization and improving the efficiency of data synchronization. By using the data conversion rules in the data synchronization strategy, the data that has passed the verification is converted into a format, and data synchronization can be performed on databases with differences in format, encoding, etc., thereby further improving the efficiency of data synchronization.

[0065] According to an embodiment of the present disclosure, the method for processing data also includes: synchronizing the data in the data synchronization queue based on the table dimension by using the message middleware; when it is determined that the target node has completed receiving the data to be synchronized, verifying the received data based on the data verification rules to obtain a synchronization verification result; when the synchronization verification result indicates that the verification has passed, performing a consistency check on the received data and the data to be synchronized, wherein the consistency check operation includes a data synchronization power check and a data business logic check.

[0066] According to an embodiment of the present disclosure, the message middleware is used to realize the decoupling between system components and realize asynchronous processing during data synchronization. The data to be synchronized is written into the data synchronization queue of the message middleware, and the synchronization information is recorded in the message middleware according to the table dimension, which is used for data verification and data migration in the event of failure. The synchronization information includes the table name, record number, sending time, data table status, etc.

[0067] According to an embodiment of the present disclosure, when it is determined that the target node has completed receiving the data to be synchronized, the received data is verified based on the data verification rules, including verification of various aspects such as the data format and data integrity. When the synchronization verification result indicates that the verification has passed, a consistency check is performed on the connected data and the data to be synchronized according to the data synchronization strategy, including a check on the logical rules of the two, specifically, including a data synchronization idempotence check, a data business logic check, etc. The data synchronization idempotence check is used to ensure that even if the same operation is executed multiple times, its result remains unchanged, which is crucial to maintaining data consistency. The data business logic check is used to verify the correctness and consistency of the data according to specific business rules, and may include full comparison and difference analysis, etc.

[0068] According to the embodiments of the present disclosure, data in the data synchronization queue is synchronized through the message middleware table dimension to determine the target node's progress in receiving the data to be synchronized. When it is determined that the target node has completed receiving the data to be synchronized, data verification and consistency checks are performed to promptly discover and resolve problems in the data synchronization process, thereby improving system stability and enhancing the efficiency and quality of data synchronization.

[0069] Figure 3 An interaction diagram of a data processing method according to an embodiment of the present disclosure is schematically shown.

[0070] like Figure 3 As shown, the interaction diagram of this embodiment may include multiple target nodes such as a source database node 310, a message middleware 320, a first target node 330, and a second target node 340, wherein the message middleware 320 is deployed on a server. The source database node 310 analyzes the acquired data table to determine a data synchronization strategy corresponding to the data table, extracts data from the data table using the data synchronization strategy, obtains data to be synchronized, and writes the data to be synchronized into the data synchronization queue of the message middleware 320.

[0071] The message middleware 320 is used to synchronize the data in the data synchronization queue based on the table dimension; for any target node among the first target node 330, the second target node 340 and other multiple target nodes, when the data to be synchronized is received, the received data is verified based on the data verification rules to obtain the synchronization verification result; when the synchronization verification result indicates that the verification is passed, the received data is checked for consistency with the data to be synchronized. When it is determined that the data received by the target node is inconsistent with the data to be synchronized, the synchronized data is repaired according to the fault repair rules corresponding to the data business logic.

[0072] The data synchronization strategy is determined by analysis, the data to be synchronized is extracted, and the message middleware 330 is used for efficient synchronization, and the data integrity is ensured through verification and consistency checks. When the data is inconsistent, fault repair is automatically triggered to reduce human intervention and improve the data synchronization efficiency of the system.

[0073] According to an embodiment of the present disclosure, the data processing method also includes: monitoring each data synchronization queue in the message middleware to obtain queue attribute information, wherein the queue attribute information includes queue availability status, message accumulation number and message processing efficiency; parsing the configured synchronization verification rules to obtain executable parameters; running the executable parameters to verify the queue attribute information; and when it is determined that there is an abnormality in the queue attribute information, logically isolating the data synchronization queue where the abnormality occurs to reject new data writing.

[0074] According to an embodiment of the present disclosure, each data synchronization queue in the message middleware corresponds to a set of source database nodes and target nodes. The queue availability status is used to characterize whether the data synchronization queue can work normally. The number of message accumulation is used to characterize the number of unprocessed messages in the queue. The message processing efficiency involves indicators such as throughput, latency, and transmission rate of the data synchronization queue.

[0075] According to an embodiment of the present disclosure, a synchronization check rule represents a precondition for executing data synchronization. An executable parameter represents a parameter in a precondition for executing data synchronization. For example, a synchronization check rule includes that the queue availability state is communicable, the number of message accumulations is less than a first threshold, and the message processing efficiency is greater than a second threshold. Among them, "communicable", "first threshold", and "second threshold" can be understood as corresponding executable parameters.

[0076] According to an embodiment of the present disclosure, each data synchronization queue in the message middleware is monitored to obtain queue attribute information, the configured synchronization verification rules are parsed to obtain executable parameters, and the executable parameters are imported into the task of verifying the queue attribute information to verify the queue attribute information. When it is determined that there is an abnormality in the queue attribute information, the data synchronization queue where the abnormality occurs is logically isolated to refuse new data writing.

[0077] For example, if data is found to be accumulated in a data synchronization queue, that is, the number of accumulated messages in the data synchronization queue is above the threshold, the data queue can be judged to be abnormal according to the synchronization verification rules, and the abnormal data queue can be logically isolated.

[0078] According to the embodiments of the present disclosure, in the process of implementing data synchronization using the message middleware, the queue attributes of the data synchronization queue are monitored, executable parameters are obtained based on the configured synchronization verification rules, the queue attribute information is verified, and the abnormal data synchronization queue is isolated in time, thereby reducing the impact of the data synchronization queue anomaly on the entire data synchronization process and improving the data synchronization efficiency.

[0079] According to an embodiment of the present disclosure, the synchronized data is repaired according to the fault repair rules corresponding to the data business logic, including: when it is determined that there is a missing second-category data in the received data, based on the data business logic, the missing second-category data is supplemented according to the first-category data to obtain the supplemented data; determining the available status of the data synchronization queue where the supplemented data is located; when it is determined that the data synchronization queue is unavailable, the supplemented data is allocated to the remaining data synchronization queues associated with the data business logic to complete the repair of the missing second-category data.

[0080] According to an embodiment of the present disclosure, the integrity of the data received by the destination is verified to determine whether the data received by the destination node is missing. For example, the number of data rows of the data to be synchronized and the data received by the destination node can be used for verification. In the case of determining that the second type of data is missing in the data received by the destination node, the missing second type of data is supplemented according to the first type of data by logical reasoning or calculation methods to ensure the integrity and consistency of the data.

[0081] According to an embodiment of the present disclosure, before sending the supplementary data to, the availability status of the data synchronization queue is checked, including whether the queue exists, whether it is reachable, whether it has access rights, etc. If the data synchronization queue where the supplementary data is located is available, the supplementary data is used to supplement the data synchronization queue where the supplementary data is located.

[0082] According to an embodiment of the present disclosure, when it is determined that a data synchronization queue is unavailable, the data service logic association is analyzed to determine the alternative data synchronization queue of the data synchronization queue where the completion data is located, and the completion data is allocated to the alternative data synchronization queue.

[0083] According to an embodiment of the present disclosure, a suitable candidate data synchronization queue is selected to receive the supplementary data according to the importance of the data and the queue attributes of the candidate data synchronization queue. Specifically, the supplementary data is allocated to the candidate data synchronization queue with a lower queue usage rate to complete the repair of the missing second-class data.

[0084] According to the embodiments of the present disclosure, through this process, the problem of missing two types of data can be effectively handled to ensure the integrity of the data, while avoiding directly restarting the data synchronization process, thereby improving the reliability and efficiency of data synchronization.

[0085] Figure 4 The flowchart of repairing synchronization data according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 4 As shown, the data to be synchronized includes data a, data b, data c, data d, and data e, wherein data a, data b, and data d are first-class data, and data c and data e are second-class data. The data received by the target node includes at least first-class data, namely data a, data b, and data c. When it is determined that the second-class data is missing in the received data, namely, when any one or more of data c and data e is missing, based on the data business logic, the missing second-class data is supplemented according to the first-class data to obtain supplemented data, namely, data a, data b, data c, data d, and data e.

[0087] Specifically, when data e is missing, data c and data d that have data business logic with data e can be used to supplement data e. When data c is missing, data a and data b that have data business logic with data c can be used to supplement data c. When data c and data e are missing, data a and data b can be used to supplement data c, and data c and data d can be used to supplement data e according to the data business logic of data a, data b, data c, data d, and data e.

[0088] Using the first type of data to repair the second type of data ensures the integrity and consistency of the data. Even if data is missing during the transmission process, the missing data can be supplemented through business logic, which enhances the system's fault tolerance and improves the reliability and efficiency of data synchronization.

[0089] According to an embodiment of the present disclosure, the data processing method also includes: when it is determined that a type of data is missing in the received data, the data received by the target node is discarded, and the data synchronization status of the target node is updated to synchronization failure; when it is determined that the data received by the target node is consistent with the data to be synchronized, the data to be synchronized is registered and updated in the target node.

[0090] According to an embodiment of the present disclosure, when it is determined that a type of data is missing in the received data, the record of the data synchronization queue is sent to the operation and maintenance personnel to prompt that a type of data in the data is missing, and the type of data and the type of data received by the target data node are discarded.

[0091] According to an embodiment of the present disclosure, when it is determined that the data received by the target node is consistent with the data to be synchronized, that is, no error or loss occurs during the transmission process, the data to be synchronized is formally recorded in the database or data storage system of the target node.

[0092] According to the embodiments of the present disclosure, by discarding a class of incomplete data and marking synchronization failure, the inconsistency problem caused by partial data update can be avoided and the accuracy of system data can be protected. At the same time, the registration and update of consistent data ensures the reliability of data synchronization, so that the data of the target node is always synchronized with the source database node, reducing the propagation of data errors and improving the efficiency of data synchronization.

[0093] According to an embodiment of the present disclosure, the data processing method also includes: in response to a received update request for a data synchronization policy, performing security maintenance on the new data synchronization policy in the update request; publishing and implementing the new data synchronization policy after maintenance to replace the original data synchronization policy.

[0094] According to an embodiment of the present disclosure, an update request is received from an administrator or an automated system, and the update request includes new data synchronization policy information. A security review is performed on the new data synchronization policy, including verification of the policy's legality, security checks, and compliance assessments to ensure that the new policy does not introduce security risks or violate data protection regulations. The execution of the new policy is simulated in a test environment to ensure that the new policy works as expected in actual applications and does not cause data errors or service interruptions.

[0095] According to an embodiment of the present disclosure, before updating, the data synchronization policy currently in use can be backed up so that it can be quickly rolled back if a problem occurs with the new policy. The new data synchronization policy that has been tested and verified is deployed to the production environment. This may involve updating configuration files, restarting services, or adjusting system parameters.

[0096] According to the embodiments of the present disclosure, after the new policy takes effect, the status and performance of data synchronization can be closely monitored to verify whether the data is correctly synchronized, and ensure that the actual effect of the new data synchronization policy is consistent with expectations. After the new data synchronization policy runs stably and is verified, the original data synchronization policy is gradually eliminated and replaced. By gradually replacing the original policy, a smooth transition is achieved and the risk of business interruption is reduced.

[0097] According to the embodiments of the present disclosure, by performing security maintenance on the data synchronization policy and improving the security of the data synchronization policy, the accuracy and efficiency of data synchronization are improved.

[0098] Based on the above data processing method, the present disclosure also provides a data processing device. Figure 5 The device is described in detail.

[0099] Figure 5 The structure block diagram of a data processing device according to an embodiment of the present disclosure is schematically shown.

[0100] like Figure 5 As shown, the data processing device 500 of this embodiment includes a policy determination module 510 , a data extraction module 520 , a data writing module 530 and a data repair module 540 .

[0101] The policy determination module 510 is used to analyze the acquired data table to determine the data synchronization policy corresponding to the data table. In one embodiment, the policy determination module 510 can be used to perform the operation S210 described above, which will not be described in detail here.

[0102] The data extraction module 520 is used to extract data from the data table using the data synchronization strategy to obtain the data to be synchronized, wherein the data synchronization strategy includes data extraction rules associated with the data business logic; in one embodiment, the data extraction module 520 can be used to execute the operation S220 described above, which will not be repeated here.

[0103] The data writing module 530 is used to write the data to be synchronized into the data synchronization queue of the message middleware, wherein the message middleware is used to synchronize the data in the data synchronization queue to the target node; in one embodiment, the data extraction module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0104] The data repair module 540 is used to repair the synchronized data according to the fault repair rules corresponding to the data service logic when it is determined that the data received by the target node is inconsistent with the data to be synchronized. In one embodiment, the data repair module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0105] According to an embodiment of the present disclosure, the data extraction module 520 includes: a data determination unit, a data extraction unit, a data verification unit, and a format conversion unit.

[0106] The data determination unit is used to determine the first category of data and the second category of data in the data table according to the association relationship between the data in the data business logic, wherein the first category of data is the business necessary data, and the second category of data can be inferred based on the first category of data.

[0107] The data extraction unit is used to extract the first type of data and the second type of data to obtain intermediate data.

[0108] The data verification unit is used to verify the intermediate data according to the data verification rules in the data synchronization strategy to obtain the intermediate verification result.

[0109] The format conversion unit is used to convert the format of the data that has passed the verification by using the data conversion rules in the data synchronization strategy to obtain the data to be synchronized when the intermediate verification result indicates that a type of data has passed the verification.

[0110] According to an embodiment of the present disclosure, the above-mentioned device also includes: a data synchronization module, a data verification module, and a data checking module.

[0111] The data synchronization module is used to synchronize the data in the data synchronization queue based on the table dimension using the message middleware.

[0112] The data verification module is used to verify the received data based on the data verification rules to obtain the synchronization verification result when it is determined that the target node has completed receiving the synchronization data.

[0113] The data checking module is used to perform a consistency check on the received data and the data to be synchronized when the synchronization check result indicates that the check has passed, wherein the consistency check operation includes a data synchronization power check and a data business logic check.

[0114] According to an embodiment of the present disclosure, the above-mentioned device also includes: a data monitoring module, a rule parsing module, a parameter operation module, and a data isolation module.

[0115] The data monitoring module is used to monitor each data synchronization queue in the message middleware to obtain queue attribute information, wherein the queue attribute information includes queue availability status, message accumulation number and message processing efficiency.

[0116] The rule parsing module is used to parse the configured synchronization verification rules to obtain executable parameters.

[0117] The parameter operation module is used to run the executable parameters to check the queue attribute information.

[0118] The data isolation module is used to logically isolate the data synchronization queue where the abnormality occurs when it is determined that the queue attribute information is abnormal, so as to refuse new data writing.

[0119] According to an embodiment of the present disclosure, the data repair module 540 includes: a data completion unit, a state determination unit, and a data allocation unit.

[0120] The data completion unit is used to complete the missing second-category data according to the first-category data based on the data business logic to obtain the completed data when it is determined that the second-category data is missing in the received data.

[0121] The state determination unit is used to determine the available state of the data synchronization queue where the completion data is located.

[0122] The data allocation unit is used to allocate the supplementary data to the remaining data synchronization queues associated with the data business logic when it is determined that the data synchronization queue is unavailable, so as to complete the repair of the missing second-category data.

[0123] According to an embodiment of the present disclosure, the above-mentioned device also includes: a data discarding module and a data updating module.

[0124] The data discarding module is used to discard the data received by the target node and update the data synchronization state of the target node to synchronization failure when it is determined that a type of data is missing in the received data.

[0125] The data updating module is used to register and update the data to be synchronized in the target node when it is determined that the data received by the target node is consistent with the data to be synchronized.

[0126] According to an embodiment of the present disclosure, the above-mentioned device further includes: a policy maintenance module and a policy replacement module.

[0127] The policy maintenance module is used to respond to a received request for updating the data synchronization policy and perform security maintenance on the new data synchronization policy in the update request.

[0128] The policy replacement module is used to publish and implement the new data synchronization policy that has been maintained to replace the original data synchronization policy.

[0129] According to an embodiment of the present disclosure, any multiple modules of the policy determination module 510, the data extraction module 520, the data writing module 530 and the data repair module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the policy determination module 510, the data extraction module 520, the data writing module 530 and the data repair module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the policy determination module 510, the data extraction module 520, the data writing module 530 and the data repair module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0130] Figure 6 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown.

[0131] like Figure 6As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0132] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0133] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0134] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0135] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0136] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present disclosure.

[0137] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0138] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0139] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0140] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0141] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0143] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: Analyze the acquired data table to determine a data synchronization strategy corresponding to the data table; Extracting data from the data table using the data synchronization strategy to obtain data to be synchronized, wherein the data synchronization strategy includes data extraction rules associated with data business logic; Writing the data to be synchronized into a data synchronization queue of a message middleware, wherein the message middleware is used to synchronize the data in the data synchronization queue to a target node; When it is determined that the data received by the target node is inconsistent with the data to be synchronized, the synchronized data is repaired according to a fault repair rule corresponding to the data service logic.

2. The method according to claim 1, characterized in that The extracting data from the data table by using the data synchronization strategy to obtain the data to be synchronized includes: According to the association relationship between the data in the data business logic, determine the first category of data and the second category of data in the data table, wherein the first category of data is business necessary data, and the second category of data can be inferred based on the first category of data; Extracting the first type of data and the second type of data to obtain intermediate data; Verify the intermediate data according to the data verification rules in the data synchronization strategy to obtain an intermediate verification result; When the intermediate verification result indicates that the type of data has passed verification, the format of the data that has passed verification is converted using the data conversion rules in the data synchronization strategy to obtain the data to be synchronized.

3. The method according to claim 2, characterized in that The method further comprises: Using the message middleware to synchronize the data in the data synchronization queue based on table dimensions; When it is determined that the target node has completed receiving the data to be synchronized, verifying the received data based on the data verification rule to obtain a synchronization verification result; When the synchronization check result indicates that the check has passed, a consistency check is performed on the received data and the data to be synchronized, wherein the consistency check operation includes a data synchronization power check and a data service logic check.

4. The method according to claim 2, characterized in that: The repairing the synchronized data according to the fault repair rule corresponding to the data service logic includes: When it is determined that the second type of data is missing in the received data, based on the data service logic, the missing second type of data is supplemented according to the first type of data to obtain supplemented data; Determine the availability status of the data synchronization queue where the completion data is located; When it is determined that the data synchronization queue is unavailable, the supplementary data is distributed to the remaining data synchronization queues associated with the data service logic to complete the repair of the missing second category data.

5. The method according to claim 4, characterized in that The method further comprises: When it is determined that the type of data is missing in the received data, the data received by the target node is discarded, and the data synchronization state of the target node is updated to synchronization failure; When it is determined that the data received by the target node is consistent with the data to be synchronized, the data to be synchronized is registered and updated in the target node.

6. The method according to claim 1, characterized in that The method further comprises: Monitor each data synchronization queue in the message middleware to obtain queue attribute information, wherein the queue attribute information includes queue availability, message accumulation number and message processing efficiency; Parse the configured synchronization check rules to obtain executable parameters; Running the executable parameter to check the queue attribute information; When it is determined that the queue attribute information has an abnormality, the data synchronization queue where the abnormality occurs is logically isolated to refuse new data writing.

7. The method according to claim 1, characterized in that The method further comprises: In response to a received update request for the data synchronization policy, performing security maintenance on a new data synchronization policy in the update request; The new data synchronization policy that has been maintained is published and takes effect to replace the original data synchronization policy.

8. A data processing device, characterized in that: The device comprises: A strategy determination module, used to analyze the acquired data table to determine a data synchronization strategy corresponding to the data table; A data extraction module, used to extract data from the data table using the data synchronization strategy to obtain data to be synchronized, wherein the data synchronization strategy includes data extraction rules associated with data business logic; A data writing module, used for writing the data to be synchronized into a data synchronization queue of a message middleware, wherein the message middleware is used for synchronizing the data in the data synchronization queue to a target node; The data repair module is used to repair the synchronized data according to the fault repair rules corresponding to the data service logic when it is determined that the data received by the target node is inconsistent with the data to be synchronized.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images