Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and discloses a data processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring an event system table; wherein the event system table comprises a data architecture version table, and the data architecture version table comprises to-be-processed events; executing the to-be-processed event by utilizing the central node, determining changed metadata information, and writing the changed metadata information into a data architecture version table; and controlling other nodes to obtain the changed metadata information from the data architecture version table, and updating the original metadata information of other nodes according to the changed metadata information.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] In order to ensure the metadata consistency of each computing node in a distributed database system, a metadata synchronization method based on a global operation log is currently commonly used. The core idea of ​​this method is to record the DDL command in a global operation log when executing a data definition language (DDL) operation. Then, each computing node in the distributed database system will read this global operation log periodically or asynchronously and update its local metadata table according to the DDL commands in the log.

[0003] Since each computing node needs to execute all DDL commands (pending events) in the global operation log to ensure metadata consistency, this will result in a lot of duplication of work. Especially in large distributed database systems, this repeated execution of DDL commands will significantly increase system overhead and reduce overall performance. Summary of the invention

[0004] In view of this, the present invention provides a data processing method, apparatus, computer equipment, storage medium and program product.

[0005] In a first aspect, the present invention provides a data processing method, which includes: obtaining an event system table; wherein the event system table includes a data architecture version table, and the data architecture version table includes events to be processed; utilizing a central node to execute events to be processed, determining changed metadata information, and writing the changed metadata information into the data architecture version table; controlling other nodes to obtain changed metadata information from the data architecture version table, and updating the original metadata information of other nodes according to the changed metadata information.

[0006] The data processing method provided in this embodiment introduces a central node to process pending events and determine the changed metadata information, thereby avoiding the situation where each computing node needs to execute all pending events in the global operation log.

[0007] At the same time, since each computing node does not need to repeatedly execute the operations of the events to be processed, the system overhead is also reduced. After the central node processes the changes, other nodes only need to obtain the changed metadata information from the data architecture version table and update it. This greatly reduces the consumption of network transmission and computing resources, thereby improving the overall performance of the system.

[0008] In an optional embodiment, the event system table also includes: a central node table; wherein, before using the central node to execute the event to be processed, determining the changed metadata information, and writing the changed metadata information into the data architecture version table, the method also includes: detecting whether the central node table is empty; if the central node table is empty, obtaining the unique identifier and campaign time period of each node among all current nodes; based on the unique identifier and campaign time period of each node, determining the central node and other nodes through a unique key.

[0009] The data processing method provided in this embodiment can detect whether the central node table is empty, so that the system can promptly discover the missing of the central node, thereby triggering the election process to select a new central node. This avoids the problem of system instability or malfunction due to the failure or absence of the central node. In addition, after the central node is elected, the node is responsible for executing the pending events and determining the metadata information of the changes. Since the central node is unique, it can ensure the consistency and accuracy of the data architecture changes, and avoid conflicts or inconsistencies caused by multiple nodes executing changes at the same time.

[0010] In an optional embodiment, the event system table also includes a node feedback table; wherein the method further includes: sending the updated original metadata information to the node feedback table; repeatedly executing the step of obtaining pending events in the data architecture version table to control other nodes to obtain changed metadata information from the data architecture version table, and updating the original metadata information according to the changed metadata information to obtain the target metadata information, until there are no pending events in the data architecture version table.

[0011] The data processing method provided in this embodiment can ensure that each event to be processed in the data architecture version table is properly processed by repeatedly executing the steps from obtaining the event to be processed to updating the target metadata information. This avoids data inconsistency or errors caused by missing events.

[0012] In an optional embodiment, controlling other nodes to obtain changed metadata information from the data architecture version table includes: controlling other nodes to obtain changed metadata information from the data architecture version table at a preset time interval; or, when the central node is used to execute an event to be processed and the changed metadata information is determined, sending a message of the changed metadata information to other nodes through a message notification module, so that the other nodes obtain the changed metadata information from the data architecture version table.

[0013] The data processing method provided in this embodiment can periodically check the change metadata information in the data architecture version table by setting a preset time interval, ensuring that other nodes can obtain the latest data changes in a timely manner. This method helps to maintain the consistency of data between various nodes and avoid decision-making errors or operational errors caused by data delays.

[0014] In addition, when the central node executes the pending event and determines the changed metadata information, the change information is immediately sent to other nodes through the message notification module. This method ensures that other nodes obtain the latest data changes in the first place, improving the real-time and accuracy of data updates.

[0015] In an optional embodiment, the method also includes: if the central node table is not empty, detecting whether the election time period of the current central node in the central node table has expired; if the election time period of the current central node in the central node table has expired, executing the steps of obtaining the unique identifier and election time period of each node among all current nodes, and determining the central node and other nodes through a unique key based on the unique identifier and election time period of each node.

[0016] The data processing method provided in this embodiment can detect whether the central node has expired, so that the system can promptly discover and handle potential problems caused by the failure of the central node. This avoids system stagnation or data inconsistency caused by central node failure or timeout failure to respond. In addition, according to the node's election time period and unique identifier, the system can dynamically adjust the selection of the central node. This helps to optimize resource allocation and ensure that an optimal central node can be selected.

[0017] In an optional embodiment, the event system table also includes a node survival heartbeat table; wherein, when the central node is used to execute the event to be processed and the changed metadata information is determined, the message of the changed metadata information is sent to other nodes through the message notification module, including: obtaining the heartbeat information sent by the central node and other nodes respectively according to a preset period; writing the heartbeat information into the node survival heartbeat table; when the central node is used to execute the event to be processed and the changed metadata information is determined, the target other nodes in the other nodes are determined from the node survival heartbeat table; and the message of the changed metadata information is sent to the target other nodes through the message notification module.

[0018] The data processing method provided in this embodiment can monitor the status of each node in real time, including the node's survival status, load status, etc., by sending heartbeat information at a preset period. This helps to timely discover and handle potential node failures or performance issues.

[0019] In a second aspect, the present invention provides a data processing device, which includes: an event system table acquisition module, used to acquire an event system table; wherein the event system table includes a data architecture version table, and the data architecture version table includes events to be processed; an execution module, used to execute the events to be processed using a central node, determine the changed metadata information, and write the changed metadata information into the data architecture version table; an update module, used to control other nodes to acquire the changed metadata information from the data architecture version table, and update the original metadata information of other nodes according to the changed metadata information.

[0020] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof.

[0022] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a schematic diagram of the interaction between a central node and other nodes through an event system table provided by an embodiment of the present invention;

[0025] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention;

[0026] Figure 3 is a flow chart of another data processing method according to an embodiment of the present invention;

[0027] Figure 4 is a structural block diagram of a data processing device according to an embodiment of the present invention;

[0028] Figure 5It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the interaction between the central node and other nodes through the event system table provided by an embodiment of the present invention.

[0031] First, check whether there are pending events that need to be executed in the event system table. If there are pending events that need to be executed in the event system table, the central node can execute the pending events, and then send the metadata information updated in each process of the central node processing the pending events to the event system table in stages. When there is updated metadata information (such as metadata version information) in the event system table, pull the updated metadata version information in the data architecture version table from the event system table and then send it to other nodes. Other nodes are updated through the updated metadata information and feedback the update completion message to the event system table. In this process, when the central node processes the pending events and obtains the updated metadata information, it can also directly send the updated metadata information to other nodes through the communication module. When the event system table receives the message that other nodes have completed the update, it can continue to read other pending events in the event system table and execute them by the central node.

[0032] As an example, there is a data definition language (DDL) that needs to be executed in the event system table. When the central node processes the DDL, the version corresponding to the DDL has been updated, and the updated version information and other information used to modify the metadata are sent to the event system table. Other nodes read from the event system table that the metadata schema version has changed and is higher than the locally cached version. In this case, the latest metadata is pulled from the engine, and at the same time, the event system table is used to tell the central node that the current stage is consistent. The central node can then perform the next step based on the situation.

[0033] In a possible implementation, the database node to which the user is connected (which can be a central node or other node) writes the pending event DDL into the event system table. When the database node is a central node, the central node directly executes the DDL, then updates the DDL to a completed state, and then feeds back the completed state to the user. When the database node is any other node, it asynchronously waits for the central node to complete execution, then updates the DDL to a completed state, and then the database node connected to the user feeds back the completed state to the user.

[0034] According to an embodiment of the present invention, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] In this embodiment, a data processing method is provided, which can be used in computer equipment, such as computers, servers, etc. Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0036] Step S201, obtaining an event system table; wherein the event system table includes a data architecture version table, and the data architecture version table includes events to be processed.

[0037] The event system table can be a system table for DDL. The event system table can include: a data schema version table (Schema version table). The data schema version table includes: version ID, DDL unique ID, DDL statement, type, and the current execution stage of the central node executing DDL. The data schema version table is used to record the DDL execution process. Each different stage of DDL execution has corresponding data schema version information. Each database node obtains the current DDL and data schema version from the table.

[0038] Step S202, using the central node to execute the event to be processed, determining the changed metadata information, and writing the changed metadata information into the data architecture version table.

[0039] The central node can represent a node used to execute pending events and determine the metadata information of the changes. The metadata information of the changes can represent the metadata update information generated by the data architecture change. Specifically, it can include the definition of new fields, the deletion mark of old fields, the change of field types, the version type, etc. Specifically, when the central node reads the pending events, it will first check the change request recorded in the data architecture version table. The change request may include modifications to the data structure, such as adding new fields, deleting existing fields, changing field types, or adjusting table structures. The central node will perform the data architecture change operation corresponding to the pending events. The operation may involve directly modifying the database schema, using a database migration tool to apply a change script, or updating the data structure by calling a specific API.

[0040] Step S203, control other nodes to obtain the changed metadata information from the data architecture version table, and update the original metadata information of other nodes according to the changed metadata information.

[0041] Other nodes retrieve the changed metadata information from the data schema version table periodically or on demand, and update their own original metadata information accordingly.

[0042] As an example, other nodes can retrieve the changed metadata information from the data architecture version table. When other nodes retrieve the changed metadata information, they update their own original data information according to the changed metadata information.

[0043] As an example, the central node may also directly send the changed metadata information to other nodes, and other nodes update their own original data information according to the changed metadata information.

[0044] The data processing method provided in this embodiment introduces a central node to process pending events and determine the changed metadata information, thereby avoiding the situation where each computing node needs to execute all pending events in the global operation log.

[0045] At the same time, since each computing node does not need to repeatedly execute the operations of the events to be processed, the system overhead is also reduced. After the central node processes the changes, other nodes only need to obtain the changed metadata information from the data architecture version table and update it. This greatly reduces the consumption of network transmission and computing resources, thereby improving the overall performance of the system.

[0046] In a possible implementation, the event system table further includes: a central node table; before step S202, the method further includes:

[0047] Step a1, check whether the central node table is empty.

[0048] Step a2: If the central node table is empty, obtain the unique identifier and election time period of each node among all current nodes.

[0049] The central node table is used to represent a table that records the current central node information. The current central node information may include: the unique identifier of the current central node, the election period, etc. Specifically, when the central node is running for election, the central node needs to carry a lease (such as 5 minutes, 10 minutes), that is, the central node needs to be constantly replaced, so it is necessary to continue to select the central node from other nodes except the current central node.

[0050] The unique identifier refers to the unique identifier of each node in the distributed system, which is used to distinguish different nodes. Specifically, it can be an IP address, host name, UUID, etc. The election period refers to the time interval or period for the node to participate in the central node election. When the current central node does not exist in the central node table, the unique identifiers and election period of all nodes in the current distributed system are collected.

[0051] Step a3, based on the unique identification of each node and the campaign time period, the central node and other nodes are determined by the unique key.

[0052] The unique key can be generated based on the hash value of the node unique identifier and the election period, the sorting result, or other algorithms. The purpose of the unique key is to find an "optimal" node among all nodes as the central node. All nodes participating in the election will compare their unique keys with the unique keys of other nodes and then determine the central node.

[0053] As an example, the node with the smallest unique key may be used as the central node, or the node with the largest unique key may be used as the central node, which is not specifically limited here.

[0054] As an example, start the transaction begin; check whether there is a central node currently, such as the following sql: Select * from camption_table where owner = 1; if the return is empty, it means that there is no central node, and other nodes insert their own information into the event system table. The core information includes the unique ID of the node, the election period (that is, the lease, which is valid during the lease period), and the owner (set to 1, indicating that there is already a central node). If the return is not empty, check whether the election period has expired. If it has expired, delete this record, such as executing delete from t where owner = 1; and then insert the information submitted by the node during the election.

[0055] If the return value is not empty, check whether the election time has expired. If it has not expired, it means that other nodes have succeeded in the election and the current node has failed to be selected, so execute rollback. When the node is written successfully, execute commit. Successful commit means successful election. If two nodes execute the election process at the same time, because owner=1 is written, and owner has a unique constraint, only one will be written successfully. That is, after one node succeeds in the election, the other node will receive a "unique index conflict error", that is, write failure, execute rollback, which means election failure, to ensure election uniqueness.

[0056] Step a4: if the central node table is not empty, check whether the election time period of the current central node in the central node table has expired.

[0057] If the central node table is not empty, it indicates that there is a current central node in the central node table. Then, it is necessary to determine whether the current central node is expired, that is, whether the election time period of the current central node has expired.

[0058] Step a5, if the election time period of the current central node in the central node table has expired, execute the steps of obtaining the unique identifier and election time period of each node in all current nodes, and based on the unique identifier and election time period of each node, determine the central node and other nodes through the unique key.

[0059] If the election period of the current central node has expired, then the above steps a2 to a3 can be executed, which will not be described in detail here.

[0060] As an example, a distributed system contains three nodes: NodeA, NodeB, and NodeC. Each node has a unique UUID as an identifier, and they all have an election time period (for example, once a minute).

[0061] Check the central node table and find that the table is empty, indicating that there is currently no central node.

[0062] Collect information from all nodes. Suppose we get the following data:

[0063] NodeA: UUID = 12345, election period = once per minute;

[0064] NodeB: UUID = 67890, election period = once per minute;

[0065] NodeC: UUID = 24680, election period = once per minute;

[0066] Use the unique key to determine the central node. In this example, we can simply select the node with the smallest UUID as the central node. Therefore, NodeA (UUID = 12345) will become the central node, while NodeB and NodeC will be ordinary nodes.

[0067] The data processing method provided in this embodiment can detect whether the central node table is empty, so that the system can promptly discover the missing of the central node, thereby triggering the election process to select a new central node. This avoids the problem of system instability or malfunction due to the failure or absence of the central node. In addition, after the central node is elected, the node is responsible for executing the pending events and determining the metadata information of the changes. Since the central node is unique, it can ensure the consistency and accuracy of the data architecture changes, and avoid conflicts or inconsistencies caused by multiple nodes executing changes at the same time.

[0068] In addition, by detecting whether the central node has expired, the system can promptly discover and handle potential problems caused by central node failure. This avoids system stagnation or data inconsistency caused by central node failure or timeout. In addition, based on the node's election time period and unique identifier, the system can dynamically adjust the selection of the central node. This helps optimize resource allocation and ensure that an optimal central node can be selected.

[0069] In a possible implementation, the event system table further includes a node feedback table, wherein the above method further includes:

[0070] Step c1, sending the updated original metadata information to the node feedback table.

[0071] Step c2, repeatedly execute the steps of obtaining pending events in the data architecture version table to control other nodes to obtain changed metadata information from the data architecture version table, and update the original metadata information according to the changed metadata information to obtain the target metadata information, until there are no pending events in the data architecture version table.

[0072] The data processing method provided in this embodiment can ensure that each event to be processed in the data architecture version table is properly processed by repeatedly executing the steps from obtaining the event to be processed to updating the target metadata information. This avoids data inconsistency or errors caused by missing events.

[0073] In a possible implementation, the above step S203 includes:

[0074] Step d1, according to the preset time interval, control other nodes to obtain the changed metadata information from the data architecture version table. Or,

[0075] Step d2, when the central node is used to execute the event to be processed and the changed metadata information is determined, the message of the changed metadata information is sent to other nodes through the message notification module, so that other nodes can obtain the changed metadata information from the data architecture version table.

[0076] In this embodiment, two methods for other nodes to obtain changed metadata information may be provided.

[0077] Specifically, in step d1, the changed metadata information can be sent to the data architecture version table. After the changed metadata information is sent to the data architecture version table, other nodes are controlled to obtain the changed metadata information from the data architecture version table at a preset time interval.

[0078] As an example, the preset time interval may be 5 seconds, or 6 seconds, etc., which is not specifically limited here.

[0079] In step d2, a message notification module can be configured. When the central node executes the event to be processed and determines the changed metadata information, the message notification module can directly send the message of the changed metadata information to other nodes, so that other nodes can obtain the changed metadata information from the data architecture version table.

[0080] In a possible implementation, step d2 can be used as the first processing step, and step d1 can be used as the second processing step. That is, the message of the changed metadata information can be sent to other nodes first through the message notification module. Then step d1 is executed, which can avoid omissions or message sending errors that may occur during the transmission process of the message notification module.

[0081] The data processing method provided in this embodiment can periodically check the metadata information of the data architecture version table by setting a preset time interval, ensuring that other nodes can obtain the latest data changes in a timely manner. This method helps to maintain the consistency of data between various nodes and avoid decision-making errors or operational errors caused by data delays.

[0082] In addition, when the central node executes the pending event and determines the changed metadata information, the change information is immediately sent to other nodes through the message notification module. This method ensures that other nodes obtain the latest data changes in the first place, improving the real-time and accuracy of data updates.

[0083] In a possible implementation, the event system table also includes a node survival heartbeat table; wherein the above step d2 includes:

[0084] Step d21, obtaining the heartbeat information sent by the central node and other nodes according to a preset period.

[0085] Step d22, write the heartbeat information into the node survival heartbeat table; when the central node is used to execute the pending event and determine the changed metadata information, determine the target other nodes in the other nodes from the node survival heartbeat table.

[0086] The node survival heartbeat table is used to determine whether each node is alive. The central node and other nodes will write their own heartbeat information according to the preset period. If the heartbeat information does not exist or has timed out, then the node is considered to be dead. The node survival heartbeat table may include fields such as the unique ID of the node, the time to send the heartbeat, whether it is a DDL central node, etc. The preset period can be 5s, 10s, etc., and is not specifically limited here. Specifically, when the central node is used to execute pending events and determine the changed metadata information, it is necessary to determine the target other nodes in other nodes from the node survival heartbeat table, that is, to determine other surviving nodes from the node survival heartbeat table.

[0087] Step d23: Send the message of the changed metadata information to other target nodes through the message notification module.

[0088] The message notification module can send messages of changed metadata information to other target nodes.

[0089] The data processing method provided in this embodiment can monitor the status of each node in real time, including the node's survival status, load status, etc., by sending heartbeat information at a preset period. This helps to timely discover and handle potential node failures or performance issues.

[0090] Please refer to Figure 3 , Figure 3 It is a flowchart of another data processing method provided according to an embodiment of the present invention.

[0091] 1. Start registering multiple nodes as surviving nodes, and write metadata information into the nodes; wherein the metadata information includes metadata version information.

[0092] 2. Determine the central node from multiple nodes and detect whether there is unexecuted DDL in the event system table.

[0093] 3. If there is unexecuted DDL in the event system table, execute the DDL through the central node and write the executed metadata version information into the event system table.

[0094] 4. Other nodes extract metadata version information from the event system table, update the metadata cache, and send the cached metadata information to the event system table; or,

[0095] The central node sends the metadata version information directly to other nodes through the communication module. Other nodes update the metadata cache and send the cached metadata information to the event system table.

[0096] 5. The central node repeats steps 2 to 4 until there is no unexecuted DDL in the event system table.

[0097] In the present embodiment, a data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0098] This embodiment provides a data processing device, such as Figure 4 As shown, it includes: an event system table acquisition module 401, used to obtain the event system table; wherein the event system table includes a data architecture version table, and the data architecture version table includes pending events; an execution module 402, used to execute pending events using a central node, determine the changed metadata information, and write the changed metadata information into the data architecture version table; an update module 403, used to control other nodes to obtain the changed metadata information from the data architecture version table, and update the original metadata information of other nodes according to the changed metadata information.

[0099] In a possible implementation, the event system table also includes: a central node table; the above-mentioned device also includes: a first detection module, used to detect whether the central node table is empty; a node identification acquisition module, used to obtain the unique identification and campaign time period of each node among all current nodes if the central node table is empty; a node determination module, used to determine the central node and other nodes through a unique key based on the unique identification and campaign time period of each node.

[0100] In one possible implementation, the event system table also includes a node feedback table; the above-mentioned device also includes: a sending module, which is used to send the updated original metadata information to the node feedback table; a repeated execution module, which is used to repeatedly execute the steps of obtaining the pending events in the data architecture version table to control other nodes to obtain the changed metadata information from the data architecture version table, and update the original metadata information according to the changed metadata information to obtain the target metadata information, until there are no pending events in the data architecture version table.

[0101] In one possible implementation, the update module 403 includes: a first acquisition unit, used to control other nodes to obtain the changed metadata information from the data architecture version table at a preset time interval; or, a second acquisition unit, used to send a message of the changed metadata information to other nodes through a message notification module when the central node is used to execute an event to be processed and the changed metadata information is determined, so that the other nodes obtain the changed metadata information from the data architecture version table.

[0102] In a possible implementation, the above-mentioned device also includes: a second detection module, which is used to detect whether the election time period of the current central node in the central node table has expired if the central node table is not empty; an execution module, which is used to execute the steps of obtaining the unique identifier and election time period of each node in all current nodes and determining the central node and other nodes through a unique key based on the unique identifier and election time period of each node if the election time period of the current central node in the central node table has expired.

[0103] In a possible implementation, the event system table also includes a node survival heartbeat table; the second acquisition unit includes: an acquisition subunit, used to obtain heartbeat information sent by the central node and other nodes according to a preset period; a writing subunit, used to write the heartbeat information into the node survival heartbeat table; a determination subunit, used to determine the target other nodes in the node survival heartbeat table when the central node is used to execute the event to be processed and the changed metadata information is determined; a sending subunit, used to send the message of the changed metadata information to the target other nodes through the message notification module.

[0104] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0105] The data processing device in this embodiment is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0106] The embodiment of the present invention also provides a computer device having the above Figure 4 The data processing device shown.

[0107] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0108] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0109] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0110] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0112] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0113] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0114] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0115] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain an event system table; wherein the event system table includes a data architecture version table, and the data architecture version table includes pending events; Utilize the central node to execute the event to be processed, determine the changed metadata information, and write the changed metadata information into the data architecture version table; Control other nodes to obtain the changed metadata information from the data architecture version table, and update the original metadata information of other nodes according to the changed metadata information.

2. The data processing method according to claim 1, characterized in that: The event system table also includes: a central node table; wherein, before using the central node to execute the event to be processed, determining the changed metadata information, and writing the changed metadata information into the data architecture version table, the method also includes: Detecting whether the central node table is empty; If the central node table is empty, obtain the unique identifier and election time period of each node among all current nodes; Based on the unique identification of each node and the election period, the central node and other nodes are determined by a unique key.

3. The data processing method according to claim 1, characterized in that: The event system table also includes a node feedback table; wherein the method further includes: Send the updated original metadata information to the node feedback table; Repeat the steps of obtaining pending events in the data architecture version table to control other nodes to obtain changed metadata information from the data architecture version table, and update the original metadata information according to the changed metadata information to obtain the target metadata information, until there are no pending events in the data architecture version table.

4. The data processing method according to claim 1, characterized in that: The metadata information that controls other nodes to obtain changes from the data architecture version table includes: At preset time intervals, control other nodes to obtain the changed metadata information from the data architecture version table; or, When the central node is used to execute the event to be processed and the changed metadata information is determined, the message of the changed metadata information is sent to other nodes through the message notification module, so that other nodes obtain the changed metadata information from the data architecture version table.

5. The data processing method according to claim 2, characterized in that: The method further comprises: If the central node table is not empty, detecting whether the election time period of the current central node in the central node table has expired; If the election time period of the current central node in the central node table has expired, execute the steps of obtaining the unique identifier and election time period of each node in all current nodes and determining the central node and other nodes through the unique key based on the unique identifier and election time period of each node.

6. The data processing method according to claim 4, characterized in that: The event system table also includes a node survival heartbeat table; wherein, when the event to be processed is executed by the central node and the changed metadata information is determined, a message of the changed metadata information is sent to other nodes through a message notification module, including: Obtain the heartbeat information sent by the central node and other nodes according to the preset period; Writing the heartbeat information into the node survival heartbeat table; When the event to be processed is executed by the central node to determine the changed metadata information, the target other node in the other nodes is determined from the node survival heartbeat table; Through the message notification module, the message of the changed metadata information is sent to other target nodes.

7. A data processing device, characterized in that: The device comprises: An event system table acquisition module, used to acquire an event system table; wherein the event system table includes a data architecture version table, and the data architecture version table includes events to be processed; An execution module, used to execute the event to be processed using the central node, determine the changed metadata information, and write the changed metadata information into the data architecture version table; The update module is used to control other nodes to obtain the changed metadata information from the data architecture version table, and update the original metadata information of other nodes according to the changed metadata information.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data processing method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the data processing method according to any one of claims 1 to 6.

Images