Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and the method comprises the steps: a global management node receives a parameter change request, updates a global parameter according to the parameter change request, generates a parameter change log, and stores the parameter change log in a parameter change log table; based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task to a parameter synchronization task table; according to the parameter synchronization task, the global management node distributes the parameter change data to the target node; the target node comprises a micro-service management node or a distributed control node; and the target node updates the parameter change data to a local parameter table. According to the scheme, parameter consistency is ensured through centralized version management, and business conflicts are avoided; a differential distribution strategy is adopted to balance the synchronization efficiency of centralized and distributed nodes, and time delay unification is realized; and log recording and standardized task scheduling are combined, so that parameter change is traceable, the problem checking efficiency is improved, and the system stability is remarkably enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of technology, the complexity of core business systems has been continuously increasing, and their architectures are gradually evolving towards microservices. The fourth generation of core business systems includes multiple types of microservice management nodes and supports a hybrid deployment of centralized and distributed modes. During the operation of the system, each microservice needs to share a large number of public parameters, and at the same time, some microservices also need to manage their own specific local parameters. In the prior art, parameter synchronization mainly faces the following challenges: First, there is insufficient compatibility in the hybrid deployment environment. The node structures of centralized microservices and distributed microservices are quite different, and traditional synchronization methods are difficult to uniformly adapt, often resulting in problems such as low synchronization efficiency of centralized nodes and high synchronization latency of distributed nodes; Second, the version management mechanism is lacking. Each microservice management node lacks a unified version identifier, and it is impossible to quickly verify the version consistency after parameter changes, which easily leads to some nodes using old version parameters and causing business logic conflicts; Third, the traceability and controllability of the synchronization process are insufficient. Parameter change operations are not fully recorded, and the generation and execution of synchronization tasks lack a standardized process, making it difficult to locate specific faulty nodes or operation steps during problem troubleshooting. Summary of the Invention

[0003] The present disclosure provides a data processing method, apparatus, device, and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, there is provided a data processing method, the method including: A global management node receives a parameter change request, updates global parameters according to the parameter change request, generates a parameter change log, and stores it in a parameter change log table; Based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in a parameter synchronization task table; According to the parameter synchronization task, the global management node distributes parameter change data to target nodes; the target nodes include microservice management nodes or distributed control nodes, and the parameter change data includes a parameter change log table and a global version number; The target node updates the parameter change data to a local parameter table.

[0005] In an implementable embodiment, based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in the parameter synchronization task table, including: The global management node polls the parameter change log table to determine whether there is a parameter change log record to be processed; When there is a parameter change log record to be processed, the global management node generates a parameter synchronization task with the single synchronization quantity and records the parameter synchronization task in the parameter synchronization task table.

[0006] In an implementable embodiment, according to the parameter synchronization task, the management node distributes parameter change data to target nodes, including: The global management node issues the parameter change data to the microservice system configured in the parameter synchronization node information table; If the microservice system is distributedly deployed, the parameter change data is issued by the global management node to the microservice management node, and the microservice management node further distributes the parameter change data to its subordinate distributed control nodes; If the microservice system is centrally deployed, the parameter change data is directly issued by the global management node to the microservice management node.

[0007] In an implementable embodiment, after the target node updates and synchronizes the parameter change data to the local parameter table, when the parameter is a global cache, the method further includes: The target node generates a new parameter version number and simultaneously updates the mapping relationship between the global version number and the local version number; After the target node updates the parameter change data to the local parameter table, all target nodes load the parameter into the memory and perform version switching according to the mapping relationship.

[0008] In an implementable embodiment, all target nodes loading the parameter into the memory includes: The target node polls to detect the parameter version upgrade status; The instance that competes for the lock loads the latest parameter into the memory and updates the status in the instance parameter version information table; The instance that does not compete for the lock obtains the latest version number according to the parameter version upgrade status and completes the memory loading.

[0009] In an implementable embodiment, performing version switching according to the mapping relationship includes: After all target nodes are successfully loaded, the superior management node of the target node issues a version switching instruction; After the superior management node confirms that all target nodes have been successfully switched, the status of the parameter synchronization log table and the parameter synchronization task table is updated.

[0010] In an implementable manner, the superior management node of the target node issues a version switching instruction, including: If the microservice system is distributedly deployed, the microservice management node issues a version switching instruction to the distributed control node, and after the distributed control node completes the switching, it reports the completion status to the microservice management node; If the microservice system is centrally deployed, the microservice management node directly executes the version switching and reports the completion status to the global management node.

[0011] According to a second aspect of the present disclosure, a data processing device is provided, and the device includes: A first generation module, configured to receive a parameter change request by the global management node, update global parameters according to the parameter change request, generate a parameter change log, and store it in a parameter change log table; A second generation module, configured to generate a parameter synchronization task by the global management node based on the parameter change log, and record the parameter synchronization task in a parameter synchronization task table; A distribution module, configured to distribute parameter change data to a target node by the global management node according to the parameter synchronization task; the target node includes a microservice management node or a distributed control node, and the parameter change data includes a parameter change log table and a global version number; An update module, configured to update the parameter change data to a local parameter table by the target node.

[0012] According to a third aspect of the present disclosure, an electronic device is provided, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the present disclosure.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to cause the computer to execute the method of the present disclosure.

[0014] The data processing method, apparatus, device, and storage medium of the present disclosure receive a parameter change request through a global management node, update global parameters according to the parameter change request, generate a parameter change log, and store it in a parameter change log table; based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in a parameter synchronization task table; according to the parameter synchronization task, the global management node distributes the parameter change data to target nodes; the target nodes include a microservice management node or a distributed control node; the target nodes update the parameter change data to a local parameter table. This solution ensures parameter consistency through centralized version management, avoiding business conflicts; adopts a differential distribution strategy to balance the synchronization efficiency of centralized and distributed nodes, achieving unified latency; combines log recording with standardized task scheduling, making parameter changes traceable, improving the problem troubleshooting efficiency, and significantly enhancing the system stability.

[0015] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present disclosure will become easily understood. In the drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, where: In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0017] Figure 1 Shows the implementation process schematic of the data processing method according to the embodiment of the present disclosure Figure 1 ; Figure 2 Shows the implementation process schematic of the data processing method according to the embodiment of the present disclosure Figure 2 ; Figure 3 Shows the implementation process schematic of the data processing method according to the embodiment of the present disclosure Figure 3 ; Figure 4 Shows the implementation process schematic of the data processing method according to the embodiment of the present disclosure Figure 4 ; Figure 5 Shows the implementation process schematic of the data processing method according to the embodiment of the present disclosure Figure 5 ; Figure 6 Shows the implementation process schematic of the data processing method according to the embodiment of the present disclosure Figure 6 ; Figure 7 Shows the structural schematic diagram of the data processing apparatus according to the embodiment of the present disclosure; Figure 8 The figure shows a schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0018] To make the objectives, features, and advantages of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only a part rather than all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0019] The present disclosure provides a data processing method, as Figure 1 shown, the method includes: Step 101: The global management node receives a parameter change request, updates the global parameters according to the parameter change request, generates a parameter change log, and stores it in the parameter change log table.

[0020] In this example, the global management node is the only node in the entire system responsible for managing the global parameters shared by all microservice systems (including centralized microservice systems and distributed microservice systems). The parameter change request originates from a visualization management system, such as web counter operations. When the global management node receives a parameter change request and performs database operations such as adding, deleting, or modifying the parameter table, a parameter change log is generated. The parameter change log specifically records the specific change content for the predefined parameter table, such as adding or modifying certain parameter values, and then stores it in the parameter change log table. By recording the changes, it provides an accurate basis for subsequent synchronization, ensures that each node can execute the same change operations, thereby ensuring the traceability of parameter changes and avoiding synchronization deviations.

[0021] Step 102: Based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in the parameter synchronization task table.

[0022] In this example, the global management node generates a parameter synchronization task by polling the parameter change log table. The types of parameter synchronization tasks are divided into global cache and non-global cache, and the two are processed alternately. The system preferentially processes non-global cache tasks when it is first started. The generated tasks are recorded in the parameter synchronization task table for tracking the synchronization progress. By converting the change log into ordered tasks, it avoids the pressure on the system or data chaos caused by batch operations, improves the synchronization efficiency, and reasonably arranges the processing priorities of different types of parameters.

[0023] Step 103: According to the parameter synchronization task, the global management node distributes parameter change data to the target nodes; the target nodes include microservice management nodes or distributed control nodes, and the parameter change data includes a parameter change log table and a global version number.

[0024] In this example, the target nodes are determined according to the microservice system deployment method. Depending on whether the microservice system is centralized or distributed, they are microservice management nodes or distributed control nodes. The parameter change data includes a parameter change log table that records specific changes and a global version number. The global version number is used for version verification between multiple systems. Its initial value comes from a specific configuration and is incremented after each global synchronization. The global management node sends the data to the target nodes according to the node list configured in the parameter synchronization node information table. Through differential distribution in different deployment environments and version number binding, it ensures that each node receives the correct change data, is compatible with different deployment scenarios, and prevents inconsistent node parameter versions.

[0025] Step 104: The target nodes update the parameter change data to the local parameter table.

[0026] In this example, when the target nodes receive the parameter change data, they replay the add, delete, and modify operations in the local parameter table in the order of the change log stream. For example, write the new parameter values into the local database. By replaying in order, it ensures the correct execution of the change operations. By establishing a version mapping to establish the connection between the global and local, it provides a basis for subsequent operations, ensures the consistency between the local parameter table and the global changes, and avoids data conflicts.

[0027] The present disclosure proposes a data processing method. The global management node receives a parameter change request, updates the global parameters according to the parameter change request, generates a parameter change log and stores it in the parameter change log table; based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in the parameter synchronization task table; according to the parameter synchronization task, the global management node distributes the parameter change data to the target nodes; the target nodes include microservice management nodes or distributed control nodes; the target nodes update the parameter change data to the local parameter table. This solution ensures parameter consistency through centralized version management, avoids business conflicts; adopts a differential distribution strategy to balance the synchronization efficiency of centralized and distributed nodes, realizes unified latency; combines log recording with standardized task scheduling, makes parameter changes traceable, improves the problem troubleshooting efficiency, and significantly enhances the system stability.

[0028] In one example, based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in the parameter synchronization task table, as Figure 2 shown, including: Step 201: The global management node polls the parameter change log table to determine whether there are any pending parameter change log records.

[0029] In this example, the global management node periodically or continuously checks the parameter change log table through a polling mechanism. Its core purpose is to proactively discover parameter change operations that need to be synchronized. The parameter change log table stores all parameter change records that have completed database insert, delete, and update operations but have not yet generated synchronization tasks, and these records are marked as "pending" status (such as adding a new interest rate rule, modifying a certain handling fee ratio, etc.). The polling mechanism ensures that change operations can be captured in a timely manner, avoiding delays in subsequent synchronization processes due to delayed processing. By actively detecting changes, it provides a trigger condition for generating synchronization tasks, ensuring the timeliness of parameter synchronization.

[0030] Step 202: When there are the pending parameter change log records, the global management node generates parameter synchronization tasks with a single synchronization quantity and records the parameter synchronization tasks in the parameter synchronization task table.

[0031] In this example, when the polling discovers that there are pending parameter change records, the global management node generates parameter synchronization tasks according to the "single synchronization quantity" pre-configured in the system. For example, if it is configured to synchronize at most 10 records each time, and there are currently 15 pending records, then 10 tasks are generated (the remaining 5 are left for the next polling process), thereby controlling the load of a single synchronization and avoiding excessive system pressure. The generated tasks are recorded in the parameter synchronization task table, and each task contains information such as the specific content of the change and the target node, which is used to track the synchronization progress and verify the completion status subsequently. Through task-based management, discrete change operations are transformed into executable synchronization instructions, ensuring the orderliness and controllability of change processing.

[0032] The polling mechanism ensures the timely response to parameter changes. The control of the single synchronization quantity balances the synchronization efficiency and system resource occupancy, and the task record provides traceability for the synchronization process, ultimately enhancing the reliability and stability of parameter synchronization.

[0033] In one example, according to the parameter synchronization task, the management node distributes the parameter change data to the target nodes, such as Figure 3 shown, including: Step 301: The global management node sends the parameter change data to the microservice system configured in the parameter synchronization node information table.

[0034] In this example, the parameter synchronization node information table is a table that stores the information of microservice systems that need to participate in parameter synchronization, including key information such as the network addresses and types of each microservice system. The global management node queries this table to identify the target microservice systems that need to receive parameter change data, ensuring that the parameter change data is only sent to the configured systems and avoiding ineffective transmission.

[0035] Step 302: If the microservice system is distributedly deployed, the parameter change data is sent from the global management node to the microservice management node, and the microservice management node further distributes the parameter change data to its subordinate distributed control nodes.

[0036] In this example, a distributedly deployed microservice system usually includes multiple subordinate nodes (such as distributed control nodes, abbreviated as DCN nodes), and intermediate nodes are needed to coordinate the synchronization. The microservice management node is the only node in the distributed microservice system responsible for managing local parameters. The global management node first sends the parameter change data to this management node, which further distributes it to each subordinate DCN node. This hierarchical distribution mechanism adapts to the characteristics of a distributed system with multiple nodes that need to be coordinated, avoiding the network pressure caused by the global management node directly communicating with a large number of DCN nodes. At the same time, through the transfer of the microservice management node, the parameter synchronization consistency within the distributed system is ensured.

[0037] Step 303: If the microservice system is centrally deployed, the parameter change data is directly sent from the global management node to the microservice management node.

[0038] In this example, a centrally deployed microservice system has no independent management node, and all its functions are directly borne by a single or a few instances. Therefore, the global management node can directly send the parameter change data to the microservice management node itself without passing through intermediate nodes. This direct distribution method simplifies the synchronization process of the centralized system, reduces the delay of intermediate links, and improves the synchronization efficiency.

[0039] By accurately positioning the target through the parameter synchronization node information table and combining the differential distribution strategies for distributed and centralized deployments, it not only ensures the hierarchical coordination of the distributed system but also optimizes the synchronization efficiency of the centralized system, ultimately enhancing the parameter synchronization compatibility and reliability of the entire system in a hybrid deployment environment.

[0040] In one example, after the target node updates and synchronizes the parameter change data to the local parameter table, when the parameter is a global cache, as Figure 4 shown, the method further includes: Step 401: The target node generates a new parameter version number and simultaneously updates the mapping relationship between the global version number and the local version number.

[0041] In this example, the global cache parameters are key parameters that need to be accessed frequently. To ensure the consistency of parameter versions between different nodes, after the target node completes the update of the local parameter table, it needs to generate a new local parameter version number. At the same time, since the global version number is a unified version identifier among multiple systems, the target node will associate the newly generated local version number with the current global version number and record it in a specific storage location (such as the parameter version mapping table). The role of this mapping relationship is that in subsequent operations, the local versions of each node can be quickly located through the global version number, ensuring cross-node version verification and synchronous tracking. By generating and mapping version numbers, a version association between the global and local levels is established, providing a basis for subsequent memory loading and version switching.

[0042] Step 402: After the target node updates the parameter change data to the local parameter table, all target nodes load the parameters into memory and perform version switching according to the mapping relationship.

[0043] In this example, the global cache parameters need to be loaded into memory to meet the requirements of frequent access. After all target nodes complete the update of the local parameter table, they will identify the current latest parameter version according to the version mapping relationship established in Step 401 and load the parameters of this version into memory. After the loading is completed, the node needs to switch the parameter version used in memory from the old version to the new version to ensure that subsequent business operations directly use the latest memory parameters. By loading into memory, the access efficiency of parameters is improved, and by version switching, it is ensured that business operations use the latest parameters, avoiding business errors caused by database read latency or old version parameters.

[0044] The version number mapping ensures the consistency between the global and local versions. Loading into memory improves the access efficiency of high-frequency parameters, and version switching ensures that business operations use the latest parameters, ultimately achieving the efficient synchronization and reliable use of global cache parameters.

[0045] In one example, all the target nodes load the parameters into memory, as Figure 5 shown, including: Step 501: The target node polls to detect the parameter version upgrade status.

[0046] In this example, the target node regularly checks the instance parameter version information table through a polling mechanism. The core purpose is to sense in real time whether the parameter version has been upgraded. The parameter version upgrade is usually triggered by a synchronization task initiated by the global management node. When the local parameter table is updated, the version information will be marked as "to be upgraded". Polling detection ensures that the node can timely detect version changes and avoid the inconsistency between memory parameters and database parameters caused by delayed loading. By actively detecting the version status, a trigger condition for subsequent memory loading is provided to ensure the timeliness of parameter updates.

[0047] Step 502: The instance that has acquired the lock loads the latest parameters into memory and updates the status in the instance parameter version information table.

[0048] In this example, to avoid resource competition or data conflicts caused by multiple instances loading the same parameters simultaneously, the system adopts a "lock mechanism": only the instance that has acquired the lock is authorized to perform the actual memory loading operation. This instance reads the latest version of the parameters from the local parameter table and loads them into memory. After the loading is completed, this instance updates the status in the instance parameter version information table, records the currently loaded version number, and provides a reference for other instances. Through the lock mechanism, data chaos caused by parallel loading of multiple instances is avoided, and at the same time, the loading result is passed through the status update.

[0049] Step 503: The instance that has not acquired the lock obtains the latest version number according to the status upgrade of the parameter and completes the memory loading.

[0050] In this example, the instance that has not acquired the lock does not need to repeat the loading operation. Instead, it directly obtains the latest version number by querying the updated status in the instance parameter version information table. These instances will extract the latest parameters from the multi-version copies already loaded in memory based on this version number (or synchronize the memory data through other efficient methods) to complete their own memory loading. This not only avoids resource waste but also ensures that all instances can ultimately use the latest parameters while maintaining business continuity.

[0051] In one example, the version switching according to the mapping relationship is as Figure 6 shown and includes: Step 601: After all target nodes have been successfully loaded, the superior management node of the target node issues a version switching instruction.

[0052] In this example, the prerequisite for all target nodes to complete parameter loading for version switching is that only when all target nodes have loaded the latest parameters into memory can the version be switched uniformly to avoid business logic chaos caused by some nodes using old version parameters. The superior management node (such as the microservice management node in a distributed microservice system or the global management node in a centralized microservice system) undertakes the coordination responsibility at this stage. By issuing a version switching instruction, it triggers all target nodes to synchronously perform the version switching operation. For example, in a distributed system, the microservice management node sends an instruction of "switch to version V2" to each DCN node under it; in a centralized system, the global management node directly sends a switching instruction to the microservice management node. By unifying the instruction, it ensures that all nodes use the latest parameters at the same time point, guaranteeing the overall parameter consistency of the system.

[0053] Step 602: After the upper-level management node confirms that all target nodes have been successfully switched, update the status of the parameter synchronization log table and the parameter synchronization task table.

[0054] In this example, after the upper-level management node issues a switching instruction, it needs to receive the "switching successful" status feedback from each target node. Only when all nodes have successfully reported can the version switch be confirmed to be completed. At this time, the upper-level management node will update the parameter synchronization log table and the parameter synchronization task table. These status updates provide key basis for subsequent process tracing and problem troubleshooting. For example, it can be seen from the log table which node has failed to switch, or it can be confirmed whether the current synchronization is ultimately successful through the task table. Through status recording and verification, the closed-loop management of the version switching process is ensured, avoiding potential risks caused by some nodes not being switched.

[0055] In one example, the upper-level management node of the target node issues a version switching instruction, including: If the microservice system is distributedly deployed, the microservice management node issues a version switching instruction to the distributed control node, and the distributed control node reports the completion status to the microservice management node after the switching is completed.

[0056] In this example, for a distributedly deployed microservice system, the microservice management node serves as the coordination center within the system and is responsible for uniformly managing the subordinate distributed control nodes (DCN nodes). During version switching, the microservice management node issues a switching instruction to each DCN node to ensure that all subordinate nodes perform the switching operation synchronously. After the DCN node completes the switching, it needs to report the "switching completed" status to the microservice management node. This feedback mechanism ensures that the microservice management node can confirm whether each DCN node has been successfully switched, avoiding parameter inconsistency problems caused by some nodes not being switched.

[0057] If the microservice system is centrally deployed, the microservice management node directly performs the version switching and reports the completion status to the global management node.

[0058] In this example, for a centrally deployed microservice system, since there is no independent microservice management node, the microservice management node directly receives the instruction from the global management node and completes the version switching by itself. After the switching is completed, the microservice management node needs to directly report the "switching completed" status to the global management node, enabling the global management node to directly grasp the switching result of the centralized system without relaying through intermediate nodes.

[0059] In one example, a complete embodiment is used to elaborate on the above data processing method in detail: A. Visually maintain parameters through the web management system (counter). If the parameter maintenance transaction is sent to the global management node, go to B (if it is sent to the management node of the microservice system, go to B'). B. During the transaction process, the global management node discovers that operations of addition, deletion, or modification have occurred in the parameter table, generates corresponding parameter change logs, and stores them in the parameter change log table (tsp_param_change_log). If the parameter is a global cache, the in-memory parameters in the current version of this instance will also be updated; C. The polling thread of the global management node checks the parameter change log (tsp_param_change_log) to determine whether a parameter synchronization task is generated. If there is an "onprocess" record in the parameter change log table, tasks will be generated according to the records configured in the application (how many records are synchronized at a time. If the number is less than the configured number, one task will be generated; if it is more than the configured number, only the configured number will be taken, and the remaining records will be left for the next synchronization). (The global cache and non-global cache tasks are carried out alternately. When the application is just started, the non-global cache task is processed for the first time, and the synchronization records are also queried according to the task type.) A record is inserted into the parameter synchronization task table (tsp_param_sync_task), and then corresponding synchronization logs are generated and inserted into the parameter synchronization log table (tsp_param_sync_log) according to the nodes configured in the parameter synchronization node information table (tsp_param_sync_node); D. After generating the parameter synchronization task, the global management node calls the remote service according to the parameter synchronization node information table (tsp_param_sync_node), and distributes the records in the parameter change log table (tsp_param_change_log) to each microservice system node configured in the parameter synchronization node information table (tsp_param_sync_node), and the input parameter needs to carry the "global version number"; 1. If the microservice system is deployed distributively, the microservice management node needs to perform the parameter synchronization process of the microservice itself: B'. The global management node distributes the records in the parameter change log table (tsp_param_change_log) to the microservice management node, directly synchronizes the parameter change log table (tsp_param_change_log), and the microservice management node receives the "global version number". If the parameter is a global cache, the mapping relationship between the global version number and the microservice version number is inserted into the parameter version mapping table (tsp_param_version_relation); C'. The polling thread of the microservice management node checks the parameter change log table (tsp_param_change_log). If there is an "onprocess" record in the parameter change log table, parameter synchronization tasks will be generated (similar to the global management node); D', after generating the parameter synchronization task, the microservice management node calls the remote service according to the parameter synchronization node information table (tsp_param_sync_node), sends the record of the parameter change log table (tsp_param_change_log) to the DCN node configured in the parameter synchronization node information table (tsp_param_sync_node), and the input parameters need to carry the "global version number"; E', each DCN node receives the "global version number", then synchronizes the difference to the parameter table. If the parameter is globally cached, the parameter version number is upgraded and the mapping relationship between the global version number and the microservice version number is inserted into the parameter version mapping table (tsp_param_version_relation); F', if the parameter is globally cached, each DCN node loads the parameter into memory through a polling thread, otherwise it does not load; G', if the parameter is a global cache, each DCN node (including ADM) needs to ensure that it completes the parameter loading. The global cache or non-global cache will report to the microservice management node and update the instance step in the parameter version information table (tsp_instans_param_version) as "report_load_param_success-report the loading parameter as successful", and update the parameter synchronization log table (tsp_param_sync_log) corresponding to the DCN status; 2. If the microservice system is deployed in a centralized manner, there is no microservice management node: E. The microservice management node receives the "global version number" (including the global management node) and directly synchronizes the difference to the parameter table. If the parameter is globally cached, the parameter version number is upgraded and the mapping relationship between the global version number and the microservice version number is inserted into the parameter version mapping table (tsp_param_version_relation); F. If the parameter is globally cached, each instance loads the parameter into memory through a polling thread; G. For distributed microservice systems, the microservice management node needs to ensure that all DCN nodes have completed the parameter loading step; for centrally deployed microservice systems, the microservice management node needs to ensure that it has completed the parameter loading step. After the parameter loading step is completed, it is reported to the global management node and the parameter synchronization log table (tsp_param_sync_log) is updated to correspond to the status of the microservice system; H. After ensuring that all microservice systems have completed parameter loading, the global parameter management node needs to send the parameter for the second time if it is a global cache to notify each microservice management node to start switching versions. Otherwise, parameter synchronization ends: 1. If the microservice system is deployed distributively, the microservice management node is required to perform the parameter synchronization process of the microservice itself: H'. The microservice management node issues a second notice to notify each DCN node of the microservice to start switching versions; I'. Each DCN node switches versions. After ensuring that it has completed the version switch itself, it updates the instance step in the parameter version information table (tsp_instans_param_version) to "report_switch_version_success - report that the version switch is successful", reports it to the microservice management node, and updates the status of the corresponding DCN in the parameter synchronization log table (tsp_param_sync_log); 2. If the microservice system is deployed centrally, there is no microservice management node: The microservice management node performs a version switch; I. For a distributively deployed microservice system, the microservice management node needs to ensure that all DCN nodes have completed the version switch; for a centrally deployed microservice system, it needs to ensure that it has completed the version switch. After the version switch is completed, it reports to the global management node, updates the status of the corresponding microservice system in the parameter synchronization log table (tsp_param_sync_log), and updates the status of the parameter synchronization task table (tsp_param_sync_task), and the global parameter synchronization ends.

[0060] The present disclosure provides a data processing device, as Figure 7 shown, the device includes: A first generation module 701, configured to receive a parameter change request by the global management node, update global parameters according to the parameter change request, generate a parameter change log, and store it in the parameter change log table; A second generation module 702, configured to generate a parameter synchronization task by the global management node based on the parameter change log, and record the parameter synchronization task in the parameter synchronization task table; A distribution module 703, configured to distribute parameter change data to a target node by the global management node according to the parameter synchronization task; the target node includes a microservice management node or a distributed control node, and the parameter change data includes a parameter change log table and a global version number; An update module 704, configured to update the parameter change data to a local parameter table by the target node.

[0061] In one example, the second generation module 702 is specifically configured to: The global management node polls the parameter change log table to determine whether there is a parameter change log record to be processed; When there is the log record of the parameter to be processed changed, the global management node generates a parameter synchronization task with a single synchronization quantity, and records the parameter synchronization task into a parameter synchronization task table.

[0062] In one example, the distribution module 703 is specifically configured to: The global management node sends the parameter change data to the microservice system configured in the parameter synchronization node information table; If the microservice system is distributedly deployed, the parameter change data is sent from the global management node to the microservice management node, and the microservice management node further distributes the parameter change data to the subordinate distributed control nodes thereof; If the microservice system is centrally deployed, the parameter change data is directly sent from the global management node to the microservice management node.

[0063] In one example, the device further includes: a third generation module 705 and a switching module 706; after the target node updates and synchronizes the parameter change data to the local parameter table, when the parameter is a global cache: The third generation module 705 is configured to generate a new parameter version number for the target node, and simultaneously update the mapping relationship between the global version number and the local version number; The switching module 706 is configured to, after the target node updates the parameter change data to the local parameter table, load the parameter to the memory for all target nodes, and perform version switching according to the mapping relationship.

[0064] In one example, when the switching module 706 loads the parameter to the memory for all target nodes, it is specifically configured to: The target node polls and detects the parameter version upgrade status; The instance that competes for the lock loads the latest parameter to the memory and updates the status in the instance parameter version information table; The instance that does not compete for the lock obtains the latest version number according to the parameter version upgrade status and completes the memory loading.

[0065] In one example, when the switching module 706 performs version switching according to the mapping relationship, it is specifically configured to: After all target nodes are successfully loaded, the superior management node of the target node sends a version switching instruction; After the superior management node confirms that all target nodes have been successfully switched, update the statuses in the parameter synchronization log table and the parameter synchronization task table.

[0066] In one example, when the switching module 706 receives the version switching instruction sent by the superior management node of the target node, it is specifically configured to: If the microservice system is deployed distributively, the microservice management node sends a version switching instruction to the distributed control node, and after the distributed control node completes the switching, it reports the completion status to the microservice management node; If the microservice system is deployed centrally, the microservice management node directly performs version switching and reports the completion status to the global management node.

[0067] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0068] Figure 8 FIG. shows a schematic block diagram of an exemplary electronic device 800 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0069] As Figure 8 shown, the device 800 includes a computing unit 801, which can execute various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0070] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0071] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute the data processing method in any other suitable manner (e.g., by means of firmware).

[0072] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), system-on-chip systems (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0073] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0074] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0075] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0076] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0077] A computer system can include a client and a server. The client and the server are generally far apart from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0078] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitations are imposed herein.

[0079] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of this disclosure, "a plurality" means two or more, unless otherwise specifically defined.

[0080] As described above, the above are only specific embodiments of this disclosure, but the protection scope of this disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed in this disclosure can easily think of changes or substitutions, which should all be covered by the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: The global management node receives a parameter change request, updates the global parameters according to the parameter change request, generates a parameter change log and stores it in the parameter change log table; Based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in the parameter synchronization task table; According to the parameter synchronization task, the global management node distributes the parameter change data to the target nodes; the target nodes include a microservice management node or a distributed control node, and the parameter change data includes the parameter change log table and the global version number; The target node updates the parameter change data to the local parameter table.

2. The method according to claim 1, characterized in that, The step that based on the parameter change log, the global management node generates a parameter synchronization task and records the parameter synchronization task in the parameter synchronization task table includes: The global management node polls the parameter change log table to determine whether there is a parameter change log record to be processed; When there is a parameter change log record to be processed, the global management node generates a parameter synchronization task with a single synchronization quantity and records the parameter synchronization task in the parameter synchronization task table.

3. The method according to claim 2, wherein The step that according to the parameter synchronization task, the management node distributes the parameter change data to the target nodes includes: The global management node distributes the parameter change data to the microservice system configured in the parameter synchronization node information table; If the microservice system is distributed, the parameter change data is distributed by the global management node to the microservice management node, and the microservice management node further distributes the parameter change data to its subordinate distributed control nodes; If the microservice system is centrally deployed, the parameter change data is directly distributed by the global management node to the microservice management node.

4. The method according to claim 1, wherein After the target node updates and synchronizes the parameter change data to the local parameter table, when the parameter is a global cache, the method further includes: The target node generates a new parameter version number and updates the mapping relationship between the global version number and the local version number; After the target node updates the parameter change data to the local parameter table, all target nodes load the parameters into memory and perform version switching according to the mapping relationship.

5. The method according to claim 4, wherein The step that all target nodes load the parameters into memory includes: The target node polls to detect the parameter version upgrade status; The instance that competes for the lock loads the latest parameters into memory and updates the status in the instance parameter version information table; The instance that does not compete for the lock obtains the latest version number according to the parameter version upgrade status and completes the memory loading.

6. The method according to claim 4, wherein The step that performs version switching according to the mapping relationship includes: After all target nodes are successfully loaded, the superior management node of the target node issues a version switching instruction; After the superior management node confirms that all target nodes have been successfully switched, the status of the parameter synchronization log table and the parameter synchronization task table is updated.

7. The method according to claim 6, characterized in that, The step that the superior management node of the target node issues a version switching instruction includes: If the microservice system is deployed distributively, the microservice management node sends a version switching instruction to the distributed control node, and after the distributed control node completes the switching, it reports the completion status to the microservice management node; If the microservice system is deployed centrally, the microservice management node directly performs version switching and reports the completion status to the global management node.

8. A data processing device, characterized in that, The device includes: A first generation module, configured to receive a parameter change request by the global management node, update global parameters according to the parameter change request, generate a parameter change log, and store it in a parameter change log table; A second generation module, configured to generate a parameter synchronization task by the global management node based on the parameter change log, and record the parameter synchronization task in a parameter synchronization task table; A distribution module, configured to distribute parameter change data to a target node by the global management node according to the parameter synchronization task; the target node includes a microservice management node or a distributed control node, and the parameter change data includes a parameter change log table and a global version number; An update module, configured to update the parameter change data to a local parameter table by the target node.

9. An electronic device, characterized in that, Includes: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images