Data processing method, device, equipment and storage medium

Abstract

The present invention discloses a data processing method, apparatus, device and storage medium. The present invention relates to the field of big data technology. The method comprises: when a change in a changed node and / or an associated node is detected, determining a node to be started and / or a node to be deactivated associated with the change in a slave system; determining first processing data associated with the node to be started from the master system, and / or determining second processing data associated with the node to be deactivated in the slave system; updating the slave system based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the master system and the updated slave system. The technical solution of the present invention can achieve the technical effect of improving the efficiency and accuracy of task processing while ensuring the continuity of task processing.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and more particularly to a data processing method, apparatus, device, and storage medium. Background Art

[0002] In recent years, in order to reduce losses caused by system failures, information systems are usually deployed in an active-active manner, with two systems deployed equally to process data synchronously.

[0003] Currently, active-active deployment is typically based on the underlying replication technology of off-site storage devices. Two systems with identical resources are replicated, one serving as the primary system and the other as the secondary system. These systems process tasks synchronously, and when the primary system fails, the secondary system switches to processing tasks.

[0004] However, this data replication method, based on underlying replication technology, makes it impossible to view differences between data processed by the master and slave systems, making it difficult to meet the requirements for business continuity in the event of system failures. Furthermore, scheduling processing units in the system requires manual analysis and operation of the operating system to determine which processing units need to be scheduled, which affects business continuity and leads to low processing efficiency and accuracy. Summary of the Invention

[0005] The embodiments of the present invention provide a data processing method, apparatus, device and storage medium to solve the problem and achieve the technical effect of improving task processing efficiency and accuracy while ensuring task processing continuity.

[0006] In a first aspect, an embodiment of the present invention provides a data processing method, applied to a data processing system, the data processing system including a master system and a slave system, the master system including a node maintenance list and full node data in the node maintenance list, the slave system being configured to store changed nodes in the node maintenance list and associated nodes associated with the changed nodes, the method comprising:

[0007] When a change in a target processing node is detected, determining a processing node to be updated in the slave system that is associated with the change; wherein the target processing node includes the changed node and / or the associated node, and the processing nodes to be updated include a node to be started and / or a node to be deactivated;

[0008] Determining first processing data associated with the node to be activated in the master system, and / or determining second processing data associated with the node to be deactivated in the slave system;

[0009] The slave system is updated based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the master system and the updated slave system.

[0010] In a second aspect, an embodiment of the present invention further provides a data processing device configured in a data processing system, the data processing system including a master system and a slave system, the master system including a node maintenance list and full node data in the node maintenance list, the slave system being configured to store changed nodes in the node maintenance list and associated nodes associated with the changed nodes, the device including:

[0011] a processing node to be updated determining module, configured to, upon detecting a change in a target processing node, determine a processing node to be updated in the slave system that is associated with the change; wherein the target processing node includes the changed node and / or the associated node, and the processing nodes to be updated include nodes to be activated and / or nodes to be deactivated;

[0012] a processing data determining module, configured to determine, from the master system, first processing data associated with the node to be activated, and / or to determine, from the slave system, second processing data associated with the node to be deactivated;

[0013] The system update module is used to update the slave system based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the master system and the updated slave system.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method as described in any one of the embodiments of the present invention is implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any one of the embodiments of the present invention.

[0016] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the data processing method as described in any one of the embodiments of the present invention.

[0017] In an embodiment of the present invention, when a change in a change node and / or an associated node is detected, the node to be started and / or the node to be stopped associated with the change in the slave system is determined; the first processing data associated with the node to be started is determined from the main system, and / or the second processing data associated with the node to be stopped is determined in the slave system; the slave system is updated based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the main system and the updated slave system, which solves the problem in the prior art of copying data based on the underlying replication technology, resulting in affecting the continuity of business processing, low business processing efficiency and low accuracy, and realizes the configuration of change nodes according to high-priority business needs, automatically analyzing and scheduling change nodes and associated nodes on the entire link, and monitoring changes in change nodes and associated associated nodes, avoiding manual analysis and manual operations, and reducing cost consumption. Automatically find out the nodes to be started and / or the nodes to be stopped that are associated with the changes in the main system. Further, the first processing data associated with the nodes to be started can be determined from the main system, and / or the second processing data associated with the nodes to be stopped can be determined in the slave system, and then the slave system is updated based on the first processing data and / or the second processing data, so as to ensure the continuity of the slave system task processing and improve the consistency of the main system and the slave system in processing the tasks to be processed, thereby achieving the technical effect of improving the efficiency and accuracy of task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0021] Figure 3 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0022] Figure 4 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0023] Figure 5 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0024] Figure 6 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0025] Figure 7 is a structural diagram of a data processing device provided according to an embodiment of the present invention;

[0026] Figure 8 It is a structural diagram of an electronic device for implementing the data processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0028] It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures. Furthermore, in the description of the present invention, the terms "first," "second," etc. are used only to distinguish descriptions and should not be understood to indicate or imply relative importance. The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0029] Before introducing the present technical solution, an example description of the application scenario can be given first. For example, in actual applications, in order to improve the availability of the data processing system and ensure the continuity of the business, a system is usually deployed in two copies, one as the main system and the other as the slave system. At this time, it is necessary to equally deploy data processing nodes for processing the business on these two systems. Each data processing node can be used as the smallest management unit for large data processing. For example, when executing a large task, multiple small tasks such as sending, loading, unloading, and transmitting are involved. Each small task corresponds to a data processing node, so as to execute the corresponding small task based on the data processing node, and to continuously execute the entire large task. In order to ensure the continuity of processing tasks based on the main system and the slave system, the technical solution provided by the embodiment of the present invention can be adopted for implementation.

[0030] Figure 1 This is a flow chart of a data processing method provided according to an embodiment of the present invention. This embodiment is applicable to processing tasks. The method can be executed by a data processing device. The data processing device can be implemented in the form of hardware and / or software. The data processing device can be configured in a data processing system. The data processing system includes a master system and a slave system. Figure 1 As shown, the method includes:

[0031] S110 . When a change in a target processing node is detected, determine a processing node to be updated that is associated with the change in the system.

[0032] It should be noted that in order to meet the business needs of multiple scenarios while ensuring the continuity of business processing and reducing the memory consumption of the slave system, data processing nodes for data processing can be deployed equally on the slave system and the master system respectively. Then, all data processing nodes in the master system, that is, all node data, will be started for processing, and tasks can be processed normally. Some data processing nodes in the slave system will be started and the other part will be disabled. The continuity of business processing can be guaranteed by scheduling the data processing nodes in the slave system.

[0033] The slave system can be a system with the same computing resources as the master system. The computing resources can be used to represent the data processing environment, for example, they can include big data platforms, servers, smart terminals, Internet of Things devices, networks, etc. The slave system is used to store the change nodes in the node maintenance list and the associated nodes associated with the change nodes, that is, to indicate that the change nodes and associated nodes are in the startup state in the slave system. The associated nodes associated with the change nodes indicate that the change nodes and the associated nodes are on the same data processing path. A data processing path may contain multiple data processing nodes, and the nodes have an upstream and downstream relationship. For example, the data processing path J is J1->J2->J3->J4, and J1, J2, J3, and J4 are four data processing nodes. If J1 is used as the change node, then J2, J3, and J4 are all associated nodes associated with J1. The master system includes a node maintenance list and the full amount of node data in the node maintenance list. The node maintenance list can be used to store the identification of the node that needs to be maintained, which is the change node. The full node data includes all data processing nodes. It is understood that both the changed nodes maintained in the node maintenance list and the processing nodes not maintained in the node maintenance list are activated in the main system. Target processing nodes include changed nodes and / or associated nodes. Processing nodes to be updated include nodes to be activated and / or nodes to be deactivated.

[0034] In this embodiment, changes in the changed node and associated nodes can be detected by timing or conditional triggering. If the changed node changes, it indicates that the user-maintained node maintenance list has been updated, and it is considered that a change in the target processing node has been detected. If the associated node changes, it indicates that the data processing path corresponding to the changed node has been updated, and it is also considered that a change in the target processing node has been detected.

[0035] It should be noted that there may be at least two implementation methods for detecting changes in target processing nodes, and these two implementation methods may be described separately. One implementation method may be: detecting that an update has occurred in the node maintenance list. The update may include at least one of addition and deletion. For example, when it is detected that new or deleted node data exists in the node maintenance list, it is considered that a change has been detected, and the newly added or deleted node may be regarded as a changed node. Another implementation method may be: detecting that an associated node associated with a changed node of at least one data processing path has been updated. For example, when it is detected that a new or deleted associated node associated with a changed node exists on a certain data processing path, it is considered that a change in the associated node has been detected.

[0036] Based on this, it can be seen that the node that has changed may be a newly added or deleted node. If it is a newly added node, then the newly added node needs to be started in the slave system for use, and the newly added node can be used as a node to be started. In order to ensure the continuity of business processing and the accuracy of business processing, the associated node associated with the newly added node can also be used as a node to be started and started in the slave system. If it is a deleted node, then the deleted node may need to be deactivated in the slave system. If the deleted node is not on the updated data processing path, the deleted node can be used as a node to be deactivated. Accordingly, both the node to be started and the node to be deactivated can be used as nodes to be updated.

[0037] To improve the effectiveness of node maintenance, when configuring a node maintenance list, you can specify the change nodes requiring maintenance based on at least one data processing path. For example, a change node can be the terminal node directly facing a downstream node—that is, the terminal node in a data processing path. For example, if the data processing path is: load task -> ETL task -> unload task -> transfer task, the transfer task node can be considered the change node requiring maintenance. This way, when searching for associated nodes associated with a change node, you only need to search for the upstream node of the change node and use all upstream nodes as the associated nodes corresponding to the change node.

[0038] On the basis of the above scheme, after the data processing nodes for data processing are equally deployed on the slave system and the master system, it is also possible to determine which data processing nodes are to be started in the slave system and which data processing nodes are to be deactivated in the slave system based on the task attributes of the data processing nodes. Optionally, if the data processing node has a preset attribute, the node status of the data processing node in both the master system and the slave system is set to the startup state; if the task attribute of the data processing node is not a preset attribute, the node status of the data processing node in the master system is set to the startup state, and the node status in the slave system is set to the deactivated state. Among them, the task attributes can be used to characterize the configuration information of the data processing node processing task. For example, the task attributes can be task type, task number, task name, planned start time, scheduling frequency, execution command, number of retries, etc. The preset attributes can be attributes such as unloading and transmission, which can be determined by the staff according to the actual work situation and are not limited here.

[0039] For example, in a big data batch processing scenario, it is usually necessary to connect data processing nodes of various task types to generate a data processing path to complete the complete data processing and supply based on each data processing node on the data processing path. For example, data processing path A is: loading task -> ETL task -> unloading task -> transfer task, where the ETL task can be used to execute the database processing statement (such as SQL) sent by the user into the database to complete data cleaning, integration and other processing; the loading task is used to load data files into the database; the unloading task is used to unload the database table or view to the file system; the transfer task is used to transfer the data files on the file system to other servers. Tasks can be connected and triggered through events, such as the input event of the ETL task, which is configured as the output event of the loading task. Among them, events support cross-domain input and output, such as the input event of the transfer task of the main system A can be the output event of the unloading task of the slave system B. The data processing node executes according to the configured rules by default. For example, when it is set to the suspended (i.e. disabled) state, the execution is suspended. Specifically, when deploying the slave and master systems equally, a set of data processing nodes needs to be deployed to both systems. Assuming the master is A and the slave is B, the specific deployment method can be: the job names and basic configurations of the two systems are identical, except for the different systems. The loading task requires an active-active deployment, outputs local domain events, and is scheduled and runs normally. The ETL task requires an active-active deployment, outputs local domain events, depends on the local loading task, and runs normally. Since the unloading task offloads the result data to a remote file system, if an active-active deployment is in the same city, the two systems can share the file system, that is, offload the result data to the same file system. If an active-active deployment is in different locations, the two systems can be separated and independent. The corresponding unloading and transmission tasks can be deployed in two different ways. One method is to deploy an active-active deployment in the same city with a shared file system. The unloading task can be deployed in active-active on both the master and slave systems, output local and cross-domain events, and depend on the local ETL task. The unloading task is scheduled and runs normally in the master system A, but is suspended in the standby system. The transmission task only needs to be deployed in the master system A, depends on the local and cross-domain unloading tasks, and runs normally. Another method is to deploy independent file systems in a dual-active manner at different locations. The offload task can be deployed in a dual-active manner on the master and slave systems, output local domain events, rely on the local domain ETL task, and operate normally on a daily basis. The transmission task can be deployed in a dual-active manner on the master and slave systems, rely on the local domain offload task, and operate normally on a daily basis.

[0040] S120 : Determine first processing data associated with the node to be started in the master system, and / or determine second processing data associated with the node to be stopped in the slave system.

[0041] In this embodiment, if the processing nodes to be updated include a node to be started, then it means that the node to be started is in an unstarted state in the slave system at the current moment, that is, a task suspended state. However, it may be in a started state in some time period before the current moment. In this case, it is only necessary to retrieve the result data processed by the node to be started that is missing in the slave system from the master system, and the retrieved result data can be used as the first processing data. Specifically, the implementation method of determining the first processing data associated with the node to be started from the master system can be: determining the unstarted time period of the node to be started in the slave system; and determining the first processing data generated during the unstarted time period corresponding to the node to be started from the master system.

[0042] In practical applications, the duration of the pending node's inactivity in the slave system can be determined based on the current change time and the time since the pending node's most recent suspension in the slave system. Furthermore, data generated by the pending node during this inactivity period can be retrieved from the master system database and used as primary processing data. This primary processing data can be used by downstream nodes of the pending node, improving the continuity and accuracy of business processing.

[0043] In this embodiment, if the nodes to be updated include a node to be deactivated, this indicates that the node to be deactivated is currently in the active state in the slave system, i.e., the task is unsuspended. The node to be deactivated can be set to suspended to suspend task processing, and the result data generated by the node's processing can be used as the second processing data.

[0044] S130 . Update the slave system based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the master system and the updated slave system.

[0045] In this embodiment, for nodes to be started, the first processed data of the node to be started is obtained from the master system database and incrementally synchronized with the slave system. The node to be started is then enabled, meaning it is unsuspended. For nodes to be deactivated, the node can be suspended and its second processed data can be deleted from the slave system, automatically clearing data to free up database storage space. This ensures business continuity and reduces memory consumption, meeting the needs of scenarios where slave system memory is limited.

[0046] Based on this, when a pending task is received, it can be processed based on the master system and the updated slave system. For example, in an actual application, when processing pending task A based on data processing path J, J1 processes subtask A1, J2 processes subtask A2, J3 processes subtask A3, and J4 processes subtask A4, and together they complete task A.

[0047] This embodiment determines the nodes to be started and / or the nodes to be stopped that are associated with the change in the slave system when a change in the change node and / or the associated node is detected; determines the first processing data associated with the node to be started in the main system, and / or determines the second processing data associated with the node to be stopped in the slave system; updates the slave system based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the main system and the updated slave system. This solves the problem in the prior art of copying data based on the underlying replication technology, which affects the continuity of business processing and has low business processing efficiency and low accuracy. This embodiment realizes the automatic search for the nodes to be started and / or the nodes to be stopped that are associated with the change in the main system by monitoring the changes in the change nodes and the associated associated nodes in the node maintenance list, thereby reducing cost consumption. Furthermore, the first processing data associated with the node to be started can be determined from the main system, and / or the second processing data associated with the node to be deactivated can be determined in the slave system, and then the slave system can be updated based on the first processing data and / or the second processing data, so as to ensure the continuity of the slave system task processing and improve the consistency of the main system and the slave system in processing the tasks to be processed, thereby achieving the technical effect of improving the efficiency and accuracy of task processing.

[0048] Based on the above embodiment, when a change in a target processing node is detected, the processing node to be updated associated with the change in the slave system can be determined by refreshing the node information in the node maintenance list, and then determining the processing node to be filtered based on the current node maintenance list and the associated nodes to be filtered associated with the current node maintenance list. Furthermore, the processing node to be updated associated with the change is determined based on the started processing nodes and the processing nodes to be filtered in the slave system. Accordingly, the present invention proposes the following embodiments: Figure 2 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 2 As shown, the data processing method includes the following steps:

[0049] S210 . When a change in the target processing node is detected, determine a current node maintenance list corresponding to the change.

[0050] Specifically, when a change in the target processing node is detected, the node information in the node maintenance list can be refreshed to determine which changed nodes are included in the current node maintenance list, so that the nodes to be started and the nodes to be disabled can be determined based on these changed nodes.

[0051] S220 : Determine a processing node to be screened based on the current node maintenance list and the associated nodes to be screened associated with the current node maintenance list.

[0052] In this embodiment, all the change nodes in the current node maintenance list can be determined, and then the associated nodes associated with these change nodes can be used as the associated nodes to be screened. All change nodes and all associated nodes to be screened can be used as nodes to be started. By searching for data processing nodes on all data processing paths, the data processing nodes except the nodes to be started can be used as nodes to be deactivated. Accordingly, the nodes to be started and the nodes to be deactivated that are preliminarily screened out can be obtained, that is, as the nodes to be screened. For example, a new change node A1 is added, and the change node A1 and all associated nodes associated with the change node A1 can be used as the nodes to be screened. At this time, the nodes to be screened are nodes to be started.

[0053] S230 : Determine the processing nodes to be updated that are associated with the change based on the started processing nodes and the processing nodes to be screened in the slave system.

[0054] In actual applications, some nodes that are initially screened out for activation may already be activated in the slave system and do not need to be reactivated. To avoid duplication and improve the accuracy and convenience of node updates, if a node is already activated in the slave system, it will be removed. If it is not activated in the slave system, it will be treated as a node to be updated. All inactive nodes that are initially screened out can be treated as nodes to be updated.

[0055] For example, all data processing paths can be stored in a scheduling system, which can then automatically identify all associated nodes associated with a changed node based on the data processing paths. When a changed node or associated nodes change, the scheduling system can automatically analyze which nodes on the system need to be activated or deactivated, while preserving the status of other nodes. Furthermore, the changed node and all associated nodes can be automatically activated, while all other nodes are deactivated.

[0056] This embodiment, upon detecting a change in a target processing node, refreshes the current node maintenance list to obtain a current node maintenance list corresponding to the change. Based on the changed nodes in the current node maintenance list and the associated nodes to be screened associated with the changed nodes, all pending processing nodes to be activated are determined. Furthermore, based on the activated processing nodes in the slave system, the activated processing nodes in the pending processing nodes are filtered out to obtain the filtered processing nodes associated with the change to be updated. This allows subsequent processing data based on the pending processing nodes to be updated to update the slave system, thereby achieving the technical effect of improving processing convenience and reducing time consumption.

[0057] Based on the above embodiment, when updating the slave system based on the second processed data, it is possible to first determine whether the accumulated deactivation time of the node to be deactivated has reached a preset retention time. If so, the second processed data in the slave system is deleted and the slave system is updated. Accordingly, the present invention proposes the following embodiments: Figure 3 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 3 As shown, the data processing method includes the following steps:

[0058] S310: Determine a deactivation attribute corresponding to the node to be deactivated.

[0059] Among them, the deactivation attributes include the deactivation time and the retention time. The retention time can refer to the preset time for retaining the node to be deactivated, such as 15 days or 30 days. The specific time can be determined by the staff according to the actual work situation and is not limited here.

[0060] In this embodiment, when a node is determined to be deactivated, the deactivation time, such as 0, can be recorded. This deactivation time can then be accumulated to obtain the accumulated deactivation duration. It should be noted that if, after determining that a node is to be deactivated, it is necessary to re-mark the node as a startup node, the accumulated deactivation duration will be reset to zero so that the next time the node is determined to be deactivated, the accumulated deactivation duration will start from zero.

[0061] S320: Determine whether the accumulated deactivation time corresponding to the deactivation moment reaches the retention time. If so, execute S330; if not, execute S340.

[0062] In this embodiment, the accumulated deactivation time can be accumulated based on the deactivation time, and the accumulated deactivation time and the retention time can be compared in real time to determine whether the accumulated deactivation time reaches the retention time. If so, it means that it is still a node to be deactivated within the set retention time, and 330 can be executed. If not, it means that the node to be deactivated is marked as a start node within the set retention time, and 340 can be executed.

[0063] S330: Delete the second processed data from the system.

[0064] Specifically, the second processed data in the slave system database may be deleted, that is, the slave system is updated.

[0065] S340: Mark the node to be deactivated as a startup node.

[0066] Specifically, the node to be deactivated may be marked as a startup node, and the second processed data from the system database is retained.

[0067] This embodiment determines the deactivation time and retention time corresponding to the node to be deactivated. If the cumulative deactivation time corresponding to the deactivation time reaches the retention time, the second processing data is deleted from the system; if the cumulative deactivation time corresponding to the deactivation time does not reach the retention time, the node to be deactivated is marked as a start node, so that the second processing data of the node to be deactivated is retained for a period of time by setting the retention time. When the cumulative deactivation time reaches the retention time, the second processing data is deleted, and the unimportant node to be deactivated is deactivated, thereby reducing memory consumption without affecting the business. When the cumulative deactivation time does not reach the retention time, no deactivation processing is performed, thereby improving the timeliness of business processing.

[0068] On the basis of the above embodiments, when processing pending tasks based on the master system and the updated slave system, each pending task may contain one or more pending subtasks. In this case, the first verification rule can be used to verify whether the result information when the master system and the slave system synchronously process the same pending subtask is consistent, thereby ensuring the accuracy and continuity of the processing. Accordingly, the present invention proposes the following embodiments: Figure 4 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 4 As shown, the data processing method includes the following steps:

[0069] S410 , for each to-be-processed subtask, when the current to-be-processed subtask is synchronously processed based on the master system and the slave system, determining first sub-processing data generated by the master system and second sub-processing data generated by the slave system.

[0070] The subtask to be processed corresponds to a data processing node in the full node data. For example, if the subtask to be processed is a loading task, the corresponding data processing node is a node for processing the loading task.

[0071] It should be noted that to ensure business processing continuity and the ability to switch between the master and slave systems at any time, automatic, fine-grained monitoring of each pending subtask being processed simultaneously by the master and slave systems is possible. The monitoring method for each pending subtask is the same, and the following description will use any pending subtask as the current processing subtask.

[0072] In this embodiment, when the main system and the slave system synchronously process the current sub-task to be processed, the job configuration information and job running log and other information of the main system can be collected in real time as the first sub-processing data, and the job configuration information and job running log and other information of the slave system can be synchronously collected as the second sub-processing data.

[0073] S420: Perform verification on the first sub-processed data and the second sub-processed data based on the first verification rule of the full amount of node data corresponding to the current sub-task to be processed, and determine the verification result.

[0074] The sub-processing data includes information such as processing sub-task duration, node configuration information, and node status. Node configuration information may refer to information used when processing sub-tasks, such as processing tools, configuration parameters, forms, etc. Node status may refer to whether it is enabled or disabled. The first verification rule may refer to a verification method used to verify that the master and slave systems consistently process tasks corresponding to data processing nodes when they are synchronously processing tasks.

[0075] In practical applications, to improve the accuracy of monitoring the synchronization of master-slave systems, a first verification rule corresponding to each data processing node in the full set of node data can be pre-configured. When processing the current pending subtask based on a data processing node, the first verification rule corresponding to the data processing node can be called to perform verification processing on the first and second sub-processing data collected synchronously in real time, such as verifying whether the job configuration information and job operation logs of the two are consistent, and obtaining a verification result.

[0076] In this embodiment, the first verification rule includes: detecting whether the processing results in the first sub-processing data and the second sub-processing data are consistent; detecting whether the difference between the processing sub-times in the first sub-processing data and the second sub-processing data does not exceed a preset first threshold; detecting whether the node configuration information in the first sub-processing data and the second sub-processing data is consistent; detecting whether the node status of the processing node in the first sub-processing data and the second sub-processing data is consistent with the actual status.

[0077] Specifically, the first verification rule can be used to compare the processing results in the first sub-processing data and the second sub-processing data to see if they are consistent. If they are consistent, the verification result is determined to be a pass in the test; if they are inconsistent, the verification result is determined to be a fail in the test. The difference between the processing sub-times (e.g., the time it takes to complete the processing) of the first sub-processing data and the second sub-processing data can also be calculated to detect whether the difference does not exceed a preset first threshold. If it does not exceed, the verification result is determined to be a pass in the test; if it exceeds, the verification result is determined to be a fail in the test; detect whether the node configuration information in the first sub-processing data and the second sub-processing data is consistent. If it is consistent, the verification result is determined to be a pass in the test; if it is inconsistent, the verification result is determined to be a fail in the test; detect whether the node status of the processing node in the first sub-processing data and the second sub-processing data is consistent with the actual status. If it is consistent, the verification result is determined to be a pass in the test; if it is inconsistent, the verification result is determined to be a fail in the test.

[0078] For example, the monitoring program corresponding to the second verification rule can be pre-configured and deployed in a unified monitoring cluster, monitoring the master and slave systems at the same time. The monitoring program collects the job configuration information and job operation logs of the master and slave systems in real time as sub-processing data, and then performs the following monitoring based on the sub-processing data: the job lists (i.e., processing results) of the master and slave systems are consistent; the configuration information (i.e., node configuration information) of each group of jobs in the master and slave systems are consistent; the difference in completion date of each group of jobs in the master and slave systems (i.e., the difference between the processing sub-times) does not exceed the set threshold (i.e., the preset first threshold); each group of jobs in the master and slave systems is in normal scheduling (i.e., startup state) or suspended state (i.e., disabled state) according to the established mode. When the monitoring program detects that the requirements of the second verification rule are not met, an alarm is sent to the operation and maintenance personnel in real time for investigation and processing. The advantage of this setting is that it can achieve real-time and automatic monitoring of the active-active tasks processed by the master and slave systems, ensuring that the master and slave systems can be switched at any time to ensure business continuity.

[0079] This embodiment determines the first processing result generated by the master system and the second processing result generated by the slave system when synchronously processing the current subtask to be processed based on the master system and the slave system, and then performs fine-grained verification processing on the first processing result and the first processing result and the corresponding related information based on the first verification rule corresponding to the current subtask to be processed, and determines the verification result, so as to determine whether an abnormality occurs based on the verification result, and make prerequisite preparations for ensuring business continuity when subsequently switching the master-slave system, so as to realize unified monitoring and real-time verification of the master-slave system without the need for large-scale data synchronization across the network.

[0080] On the basis of the above embodiment, after the master system and the updated slave system process the pending task, the master system may further determine the first result data obtained by processing the pending task to determine the second result data obtained by processing the pending task by the slave system; the first result data and the second result data may be processed based on the second verification rule to determine a comparison result, and then, based on the comparison result, determine whether the processing by the master and slave systems is consistent. Accordingly, the present invention proposes the following embodiments: Figure 5 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 5 As shown, the data processing method includes the following steps:

[0081] S510: Determine first result data obtained by the master system processing the task to be processed, and determine second result data obtained by the slave system processing the task to be processed.

[0082] In this embodiment, after the master system and the updated slave system process the pending task, the result data obtained from processing the entire pending task can be verified to ensure that the data quality meets preset requirements. In this case, the result data output by the master system when processing the pending task and the result data output by the slave system when processing the pending task can be synchronously collected as first result data and second result data ( S520 ). The first result data and the second result data are processed based on the second verification rule to determine a comparison result.

[0083] In practical applications, the second verification rule can be automatically triggered or conditionally triggered to perform comparison processing on the first result data and the second result data, such as detecting whether the data output by the two are consistent to obtain a comparison result.

[0084] In this embodiment, the second verification rule includes: detecting whether the first result data and the second result data are consistent; detecting whether the difference between the processing time of the first result data and the second result data does not exceed the preset second threshold; detecting whether the field summation value of the first result data is consistent with the field summation value of the second result data; detecting whether the first random sampling data of the first result data is consistent with the second random sampling data of the second result data.

[0085] Specifically, the first result data and the second result data can be compared using the second inspection rule to see if they are consistent. If they are consistent, the comparison result is determined to be a pass in the test; if they are inconsistent, the comparison result is determined to be a failure in the test. The difference between the processing time of the first result data and the second result data (e.g., the time for the processing to be completed) can also be calculated to detect whether the difference does not exceed a preset second threshold value. If it does not exceed, the comparison result is determined to be a pass in the test; if it exceeds, the comparison result is determined to be a failure in the test; the first result data and the second result data can also be subjected to field summation processing respectively to calculate the field summation value of the two, and compare whether the field summation values ​​are consistent. If they are consistent, the comparison result is determined to be a pass in the test; if they are inconsistent, the comparison result is determined to be a failure in the test; the first result data can also be subjected to random sampling processing to obtain first random sampling data, and the second result data can be subjected to random sampling processing to obtain second random sampling data, and the first random sampling data and the second random sampling data can be compared to see if they are consistent. If they are consistent, the comparison result is determined to be a pass in the test; if they are inconsistent, the comparison result is determined to be a failure in the test. It should be noted that the first and second result data can be recorded using a table. For example, the second verification rule includes, but is not limited to: consistency check of the number of records in the entire table; consistency check of the number of records in the table slice date; consistency check of the sum of table fields; consistency check of table sampling; etc. The advantage of this configuration is that, by performing consistency checks on the result data of the active-active master-slave system, data quality issues are prevented after the master-slave system switchover.

[0086] This embodiment determines the first result data obtained by the main system processing the task to be processed, determines the second result data obtained by the slave system processing the task to be processed, and then processes the first result data and the second result data based on the second verification rule to determine the comparison result, so as to evaluate the quality of the result data through the comparison result and ensure the data quality.

[0087] On the basis of the above embodiments, when a fault in the main system is detected, it is also possible to determine whether to allow switching and whether an abnormality occurs based on the verification result and / or comparison result. Accordingly, the present invention proposes the following embodiments: Figure 6 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 6 As shown, the data processing method includes the following steps:

[0088] S610: When a failure of the main system is detected, a reference switching result is determined based on the verification result and the comparison result.

[0089] Among them, the fault can be a network fault, circuit fault, server fault, database fault, message fault, etc.

[0090] In this embodiment, when a failure in the master system is detected, it can be determined whether the current master-slave system can be switched. This can be determined based on the verification results and the comparison results. For example, if either the verification result or the comparison result fails, the reference switching result can be determined as not allowing the switching; if both the verification result and the comparison result pass, the reference switching result can be determined as allowing the switching.

[0091] For example, the switching program and the monitoring program can be deployed together and communicate with each other. For example, suppose the master system is operating normally and suddenly experiences an abnormal database failure. The monitoring program detects the abnormal failure and can send a message to the switching program. The switching program obtains the task operation status of the master and slave systems from the monitoring program, determines a reference switching result, and then decides whether to switch based on the reference switching result.

[0092] S620: When the reference switching result indicates that the switching is allowed, suspend all node data in the master system, and determine the node status of the suspension processing node corresponding to the suspension in the slave system.

[0093] In this embodiment, if the reference switching result is that switching is not allowed, then it means that there may be a verification inconsistency between the master and slave systems at this time, and the master and slave systems are not allowed to switch. If the reference switching result is that switching is allowed, then it means that the verification of the master and slave systems is consistent at this time, and the master and slave systems are allowed to switch. Further, all data processing nodes in the full amount of node data in the master system can be suspended for processing, or the data processing nodes associated with the fault can be suspended for processing. For example, if it is a database fault, the data processing nodes involving the use of the database can be suspended for processing. The data processing nodes suspended in the master system can be used as suspended processing nodes. It should be noted that the suspended processing nodes suspended in the master system may be in the startup state, the deactivated state, or the reserved observation state in the slave system. The node state of the suspended processing node in the slave system can be determined to determine whether the suspended processing node needs to be started based on the node state.

[0094] S630: If the node state is the reserved observation state, mark the suspended processing node as a start node.

[0095] Specifically, if the node status is "Reserve Observation," the pending processing node is identified as a pending deactivation node, and within the retention period, it will be observed whether it is promoted to a change node or an associated node. The pending processing node can be marked as a startup node. If the node status is "Deactivated," the pending processing node can be started. If the node status is "Started," no action is taken.

[0096] Exemplarily, the switching program obtains the running status of the master-slave system tasks from the monitoring program. If the reference switching result meets expectations, a message is sent to the scheduling system of the master-slave system to initiate the switching. Specifically, the relevant data processing nodes of the master system (i.e., suspended processing nodes) are suspended. For example, if the shared file system mode is deployed in the same city using active-active mode, the unloading tasks of the master system need to be suspended, and the relevant data processing nodes of the slave system need to be unsuspended. For example, if the shared file system mode is deployed in the same city using active-active mode, the unloading tasks of the slave system need to be unsuspended.

[0097] In order to improve the timeliness of the maintenance of the master-slave system, in this embodiment, if the verification result or the comparison result is that the detection fails, an early warning prompt information is generated and sent to the target terminal.

[0098] Specifically, when the monitoring program detects that the inspection result or comparison result is a failure, an early warning prompt message can be generated based on the failed detection item. Furthermore, the early warning prompt message can be sent to the early warning prompt information in a preset sending method (for example, email, SMS, message, phone call, pop-up window, etc.) to alert the operation and maintenance personnel to investigate and handle it.

[0099] This embodiment determines the reference switching result based on the verification result and the comparison result when a failure of the main system is detected, and allows the master-slave system to switch when the reference switching result is that the switching is allowed. It can suspend the full amount of node data in the main system and determine the node status of the suspended processing node corresponding to the suspension in the slave system. If the node status is a retained observation state, the suspended processing node is marked as a start node to prevent the suspended processing node from being promoted to a change node or being deactivated by the associated node beyond the retention period, thereby improving business continuity and processing accuracy.

[0100] As an alternative embodiment to the above embodiment, in order to make those skilled in the art further understand the technical solution of the embodiment of the present invention, a specific application scenario example is given. For details, please refer to the following specific content.

[0101] In actual applications, the verification program can be embedded in the scheduling task in the form of a processing script. The specific implementation method can be: after the data processing main script is executed based on the data processing node task, the active-active consistency verification script can be executed. The verification table can be verified according to the first verification rule configured in the data processing node association table; after the verification is completed, if the task of this system is completed first, that is, the verification result sent by the other system is not received, the verification result of this system needs to be sent to the other system; if the task of this system is completed later, the verification result of this system is compared with the verification result received from the other system; if the verification results are inconsistent, the scheduling is terminated and a message is pushed to the other system, which also terminates the scheduling after receiving the message. When the verification results are inconsistent, the monitoring program will also receive a message and send an alarm to the operation and maintenance personnel for investigation and processing.

[0102] This embodiment automatically and accurately detects the consistency of task processing through a verification program, thereby improving the effectiveness of system switching, and achieving the technical effect of improving task processing efficiency and accuracy while ensuring the continuity of task processing.

[0103] Figure 7 FIG. 1 is a structural diagram of a data processing device provided according to an embodiment of the present invention. Figure 7 As shown, the apparatus includes: a processing node to be updated determining module 710 , a processing data determining module 720 and a system updating module 730 .

[0104] Among them, the processing node determination module 710 to be updated is used to determine the processing node to be updated associated with the change in the slave system when a change in the target processing node is detected; wherein, the target processing node includes the changed node and / or the associated node, and the processing node to be updated includes the node to be started and / or the node to be deactivated; the processing data determination module 720 is used to determine the first processing data associated with the node to be started from the main system, and / or determine the second processing data associated with the node to be deactivated in the slave system; the system update module 730 is used to update the slave system based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the main system and the updated slave system.

[0105] The technical solution of this embodiment is to determine the nodes to be started and / or the nodes to be stopped that are associated with the changes in the slave system when a change in the change node and / or the associated node is detected; determine the first processing data associated with the nodes to be started from the main system, and / or determine the second processing data associated with the nodes to be stopped in the slave system; update the slave system based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the main system and the updated slave system, which solves the problem in the prior art of copying data based on the underlying replication technology, which affects the continuity of business processing and has low business processing efficiency and low accuracy, and realizes the automatic search for the nodes to be started and / or the nodes to be stopped that are associated with the changes in the main system by monitoring the changes in the change nodes and the associated associated nodes in the node maintenance list, thereby reducing cost consumption. Furthermore, the first processing data associated with the node to be started can be determined from the main system, and / or the second processing data associated with the node to be deactivated can be determined in the slave system, and then the slave system can be updated based on the first processing data and / or the second processing data, so as to ensure the continuity of the slave system task processing and improve the consistency of the main system and the slave system in processing the tasks to be processed, thereby achieving the technical effect of improving the efficiency and accuracy of task processing.

[0106] Based on the above device, optionally, the device further includes a change detection module, and the change detection module is used to detect changes in the target processing node.

[0107] The change detection module is specifically used to detect that the node maintenance list is updated; and / or, detect that the associated node associated with the changed node of at least one data processing path is updated; wherein, the update includes at least one of addition and deletion.

[0108] On the basis of the above apparatus, optionally, the processing node to be updated determining module 710 includes a node maintenance list refreshing unit, a processing node to be screened determining unit, and a processing node to be updated determining unit.

[0109] a node maintenance list refreshing unit, configured to, upon detecting a change in a target processing node, determine a current node maintenance list corresponding to the change;

[0110] a to-be-screened processing node determining unit, configured to determine the to-be-screened processing node based on the current node maintenance list and the to-be-screened associated nodes associated with the current node maintenance list;

[0111] The processing node to be updated determining unit is configured to determine the processing node to be updated associated with the change based on the started processing nodes in the slave system and the processing nodes to be screened.

[0112] On the basis of the above device, optionally, the processed data determination module 720 includes a non-startup duration determination unit and a first processed data determination unit.

[0113] a non-startup time length determining unit, configured to determine a non-startup time length of the node to be started in the slave system;

[0114] The first processing data determining unit is configured to determine, from the main system, first processing data generated during the non-startup time period and corresponding to the node to be started.

[0115] On the basis of the above apparatus, optionally, the system update module 730 includes a deactivation attribute determination unit, a second processed data deletion unit, and a to-be-deactivated node marking unit.

[0116] A deactivation attribute determining unit, configured to determine a deactivation attribute corresponding to the node to be deactivated; wherein the deactivation attribute includes a deactivation time and a retention duration;

[0117] a second processed data deleting unit, configured to delete the second processed data from the system if the accumulated deactivation time corresponding to the deactivation moment reaches the retention time;

[0118] The to-be-deactivated node marking unit is configured to mark the to-be-deactivated node as a startup node if the accumulated deactivation duration corresponding to the deactivation moment does not reach the retention duration.

[0119] Based on the above-mentioned device, optionally, the task to be processed includes at least one sub-task to be processed corresponding to the full amount of node data, and the device also includes a system processing module, and the system processing module includes a processing data determination unit and a verification result determination unit.

[0120] a processing data determining unit, configured to determine, for each of the subtasks to be processed, first processing data generated by the master system and second processing data generated by the slave system when the current subtask to be processed is synchronously processed based on the master system and the slave system;

[0121] A verification result determination unit is used to perform verification processing on the first processed data and the second processed data based on a first verification rule of the full amount of node data corresponding to the current subtask to be processed, and determine a verification result.

[0122] Based on the above device, optionally, the first verification rule includes:

[0123] detecting whether processing results in the first processed data and the second processed data are consistent;

[0124] detecting whether a difference between the processing sub-time lengths in the first processed data and the second processed data does not exceed a preset first threshold;

[0125] detecting whether node configuration information in the first processed data and the second processed data is consistent;

[0126] It is detected whether the node states of the processing nodes in the first processing data and the second processing data are consistent with the actual states.

[0127] On the basis of the above device, optionally, the device further includes a comparison module, and the comparison module includes a result data determination unit and a comparison result determination unit.

[0128] a result data determining unit, configured to determine first result data obtained by the master system processing the task to be processed, and determine second result data obtained by the slave system processing the task to be processed;

[0129] A comparison result determination unit is used to process the first result data and the second result data based on a second verification rule to determine a comparison result.

[0130] Based on the above device, optionally, the second verification rule includes:

[0131] detecting whether the first result data and the second result data are consistent;

[0132] detecting whether a difference between the processing time of the first result data and the processing time of the second result data does not exceed a preset second threshold;

[0133] Detecting whether a field sum value of the first result data is consistent with a field sum value of the second result data;

[0134] Check whether the first random sampling data of the first result data is consistent with the second random sampling data of the second result data.

[0135] On the basis of the above device, optionally, the device further includes: a switching module, the switching module including a reference switching result determination unit, a node state determination unit and a processing unit.

[0136] a reference switching result determining unit, configured to determine a reference switching result based on a verification result and a comparison result when a failure of the main system is detected;

[0137] a node status determining unit, configured to suspend all node data in the master system when the reference switching result indicates that the switching is permitted, and determine the node status of a suspension processing node corresponding to the suspension in the slave system;

[0138] The processing unit is configured to mark the suspended processing node as a start node if the node state is a reserved observation state.

[0139] On the basis of the above-mentioned device, optionally, the device further includes an early warning module, wherein the early warning module is used to generate an early warning prompt message if the verification result or the comparison result is that the detection fails, and send the early warning prompt message to the target terminal.

[0140] The data processing device provided by the embodiment of the present invention can execute the data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0141] Figure 8 The present invention provides a schematic structural diagram of an electronic device. Figure 8 A block diagram of an exemplary electronic device 80 suitable for implementing exemplary embodiments of the present invention is shown. Figure 8 The electronic device 80 shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0142] like Figure 8 As shown, electronic device 80 is a general-purpose computing device. Components of electronic device 80 may include, but are not limited to, one or more processors or processing units 801, system memory 802, and a bus 803 connecting various system components (including system memory 802 and processing unit 801).

[0143] Bus 803 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0144] The electronic device 80 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 80, including volatile and non-volatile media, removable and non-removable media.

[0145] The system memory 802 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 804 and / or cache memory 805. The electronic device 80 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 806 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 Not shown, often called a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 803 via one or more data medium interfaces. Memory 802 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0146] A program / utility 808 having a set (at least one) of program modules 807 may be stored, for example, in memory 802. Such program modules 807 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 807 generally implement the functions and / or methods of the embodiments described herein.

[0147] The electronic device 80 may also communicate with one or more external devices 809 (e.g., keyboard, pointing device, display 810, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 80, and / or any device that enables the electronic device 80 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 811. Furthermore, the electronic device 80 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 812. As shown, the network adapter 812 communicates with other modules of the electronic device 80 via the bus 803. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 80, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0148] The processing unit 801 executes various functional applications and data processing by running programs stored in the system memory 802, such as implementing the data processing method provided by the embodiment of the present invention.

[0149] An embodiment of the present invention further provides a storage medium containing computer-executable instructions. When the computer-executable instructions are executed by a computer processor, the computer-executable instructions are used to perform a data processing method, the method comprising:

[0150] When a change in a target processing node is detected, determining a processing node to be updated in the slave system that is associated with the change; wherein the target processing node includes the changed node and / or the associated node, and the processing nodes to be updated include a node to be started and / or a node to be deactivated;

[0151] Determining first processing data associated with the node to be activated in the master system, and / or determining second processing data associated with the node to be deactivated in the slave system;

[0152] The slave system is updated based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the master system and the updated slave system.

[0153] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0154] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0155] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0156] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] An embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the data processing method provided in any embodiment of the present application.

[0158] The computer program product may be implemented by writing computer program code for performing the operations of the present invention in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions are possible for those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: Applied to a data processing system, the data processing system includes a master system and a slave system, the master system includes a node maintenance list and full node data in the node maintenance list, the slave system is used to store changed nodes in the node maintenance list and associated nodes associated with the changed nodes, the method includes: When a change in a target processing node is detected, determining a processing node to be updated in the slave system that is associated with the change; wherein the target processing node includes the changed node and / or the associated node, and the processing nodes to be updated include a node to be started and / or a node to be deactivated; Determining first processing data associated with the node to be activated in the master system, and / or determining second processing data associated with the node to be deactivated in the slave system; The slave system is updated based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the master system and the updated slave system.

2. The method according to claim 1, characterized in that Detecting a change in the target processing node includes: detecting that the node maintenance list is updated; and / or, It is detected that an associated node associated with the changed node of at least one data processing path is updated; wherein the update includes at least one of addition and deletion.

3. The method according to claim 1, characterized in that When a change in the target processing node is detected, determining a processing node to be updated in the slave system that is associated with the change includes: When a change in the target processing node is detected, determining a current node maintenance list corresponding to the change; Determining a processing node to be screened based on the current node maintenance list and associated nodes to be screened associated with the current node maintenance list; Based on the started processing nodes in the slave system and the processing nodes to be screened, the processing nodes to be updated associated with the change are determined.

4. The method according to claim 1, wherein The determining, from the main system, first processing data associated with the node to be started includes: Determining a time period during which the node to be started has not been started in the slave system; Determine, from the main system, first processed data generated during the non-startup time period and corresponding to the node to be started.

5. The method according to claim 1, wherein Updating the slave system based on the second processed data includes: Determining a deactivation attribute corresponding to the node to be deactivated; wherein the deactivation attribute includes a deactivation time and a retention duration; If the accumulated deactivation time corresponding to the deactivation moment reaches the retention time, deleting the second processed data from the secondary system; If the accumulated deactivation duration corresponding to the deactivation moment does not reach the retention duration, the node to be deactivated is marked as a start-up node.

6. The method according to claim 1, characterized in that The pending task includes at least one pending subtask corresponding to the full amount of node data, and the processing of the pending task based on the master system and the updated slave system includes: For each of the to-be-processed subtasks, when the current to-be-processed subtask is synchronously processed based on the master system and the slave system, determining first sub-processing data generated by the master system and second sub-processing data generated by the slave system; The first sub-processing data and the second sub-processing data are checked based on a first check rule for the full amount of node data corresponding to the current sub-task to be processed to determine a check result.

7. The method according to claim 6, characterized in that The first verification rule includes: detecting whether processing results in the first sub-processed data and the second sub-processed data are consistent; detecting whether a difference between the processing sub-time lengths in the first sub-processing data and the second sub-processing data does not exceed a preset first threshold; detecting whether node configuration information in the first sub-processing data and the second sub-processing data is consistent; It is detected whether the node states of the processing nodes in the first sub-processing data and the second sub-processing data are consistent with actual states.

8. The method according to claim 1, characterized in that After the to-be-processed tasks are processed based on the master system and the updated slave system, the method includes: Determine first result data obtained by the master system processing the task to be processed, and determine second result data obtained by the slave system processing the task to be processed; The first result data and the second result data are processed based on the second verification rule to determine a comparison result.

9. The method according to claim 8, characterized in that The second verification rule includes: detecting whether the first result data and the second result data are consistent; detecting whether a difference between the processing time of the first result data and the processing time of the second result data does not exceed a preset second threshold; Detecting whether a field sum value of the first result data is consistent with a field sum value of the second result data; Check whether the first random sampling data of the first result data is consistent with the second random sampling data of the second result data.

10. The method according to claim 6 or 8, characterized in that Also includes: When a failure of the main system is detected, determining a reference switching result based on the verification result and the comparison result; When the reference switching result is that the switching is allowed, suspending all node data in the master system, and determining the node status of the suspension processing node corresponding to the suspension in the slave system; If the node state is the reserved observation state, the suspended processing node is marked as a start node.

11. The method according to claim 6 or 8, characterized in that Also includes: If the verification result or the comparison result is that the detection fails, an early warning prompt message is generated and sent to the target terminal.

12. A data processing device, characterized in that: Configured in a data processing system, the data processing system includes a master system and a slave system, the master system includes a node maintenance list and full node data in the node maintenance list, and the slave system is used to store changed nodes in the node maintenance list and associated nodes associated with the changed nodes, including: a processing node to be updated determining module, configured to, upon detecting a change in a target processing node, determine a processing node to be updated in the slave system that is associated with the change; wherein the target processing node includes the changed node and / or the associated node, and the processing nodes to be updated include nodes to be activated and / or nodes to be deactivated; a processing data determining module, configured to determine, from the master system, first processing data associated with the node to be activated, and / or to determine, from the slave system, second processing data associated with the node to be deactivated; The system update module is used to update the slave system based on the first processing data and / or the second processing data, so that when a task to be processed is received, the task to be processed is processed based on the master system and the updated slave system.

13. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 11.

Images