Data processing system, method and device, computer equipment and storage medium

Abstract

The embodiment of the invention relates to a data processing system, method and device, computer equipment and a storage medium, and the system comprises a secondary node which is used for transmitting second data to a database according to the operation state of a primary node; the first-level node is used for receiving second data sent by the second-level node when the operation state is normal operation, and sending the first data and the second data in the first-level node to the database; and the database is used for receiving the first data and the second data sent by the first-level node when the operation state is normal operation, or receiving the second data sent by the second-level node when the operation state is abnormal operation. Therefore, the mode of writing the second data in the second-level node and the first data in the first-level node into the database is determined according to the running state of the first-level node, it is guaranteed that data in each node running normally is written into the database, and when the first-level node is unavailable, writing of the data of the second-level node into the database is not affected; the efficiency of writing the node data into the database is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of distributed data processing technology, and in particular, to a data processing system, method, apparatus, computer equipment, and storage medium. Background Art

[0002] In a multi-node distributed data collection and reporting system, the role allocation of master nodes and slave nodes and reliable data transmission are the key to ensuring efficient and real-time operation of the system.

[0003] In the existing master-slave architecture, when the master node fails, the slave nodes that rely on the master node cannot complete data reporting, resulting in data backlogs and possible data writing delays in the system. If all nodes are directly connected to the database, the number of database connections will increase with the number of nodes, increasing the system load and also affecting the efficiency of data writing to the database. Therefore, how to improve the efficiency of data writing to the database for each node in the distributed system has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, in order to solve the above-mentioned technical problem of improving the efficiency of writing data of each node in a distributed system into a database, an embodiment of the present invention provides a data processing system, method, apparatus, computer equipment and storage medium.

[0005] In a first aspect, an embodiment of the present invention provides a data processing system, including: a primary node, a secondary node, and a database;

[0006] The secondary node is used to send second data to the database according to the operating status of the primary node;

[0007] The primary node is configured to receive the second data sent by the secondary node when the operation status is normal operation, and send the first data and the second data in the primary node to the database;

[0008] The database is used to receive the first data and the second data sent by the primary node when the operating status is normal operation, or to receive the second data sent by the secondary node when the operating status is abnormal operation.

[0009] In a possible implementation, the primary node includes: a first database channel, and the secondary node includes: a second reporting channel;

[0010] The second reporting channel is used to send the second data to the first database channel when the operating status is normal operation;

[0011] The first database channel is used for receiving the second data and sending the first data and the second data to the database when the operation status is normal operation;

[0012] In a possible implementation manner, the secondary node further includes: a second database channel;

[0013] The second database channel is used to obtain the second data in the second reporting channel and send the second data to the database when the operating status is abnormal operation and the new first-level node has not been determined currently.

[0014] In a possible implementation, the system further includes: a target service, configured to determine a primary node and a plurality of secondary nodes from a plurality of nodes in the distributed system;

[0015] The target service is further used to: when the primary node runs abnormally, select a target secondary node from the plurality of secondary nodes as a new primary node;

[0016] The first-level node also includes: a first reporting channel, which is used to send the first data of the first-level node to the new first database channel of the new first-level node when the first-level node recovers from abnormal operation to normal operation and the new first-level node has been determined.

[0017] In a second aspect, an embodiment of the present invention provides a data processing method, which is applied to a primary node in a distributed system, wherein the distributed system further includes a plurality of secondary nodes, and the method includes:

[0018] Acquire, through the first database channel in the first-level node, second data sent by the second reporting channel of the second-level node;

[0019] Acquire first data from a data acquisition module of the primary node;

[0020] The first data and the second data are sent to a database through the first database channel.

[0021] In a possible implementation, the method further includes:

[0022] When the operation state of the first-level node switches from abnormal operation to normal operation, determining whether a new first-level node has been determined;

[0023] If the new first-level node has been determined, the first data of the first-level node is sent to a new first database channel of the new first-level node;

[0024] If the new primary node is not currently determined, the step of acquiring the second data sent by the second reporting channel of the secondary node through the first database channel in the primary node is performed.

[0025] In a third aspect, an embodiment of the present invention provides a data processing method, which is applied to a secondary node in a distributed system, wherein the distributed system also includes a primary node, and the method includes:

[0026] Determine whether the primary node is operating normally;

[0027] When the primary node operates normally, the second data in the secondary node is sent to the first database channel of the primary node through the second reporting channel of the secondary node, so that the first database channel sends the second data and the first data in the primary node to the database;

[0028] When the primary node operates abnormally and a new primary node has not been determined currently, the second data in the second reporting channel is sent to the second database channel of the secondary node, so as to send the second data to the database through the second database channel.

[0029] In a fourth aspect, an embodiment of the present invention provides a data processing device, which is applied to a primary node in a distributed system, wherein the distributed system further includes a plurality of secondary nodes, and the device includes:

[0030] A first acquisition module, configured to acquire, through the first database channel in the first-level node, second data sent by the second reporting channel of the second-level node;

[0031] A second acquisition module, used to acquire first data from the data acquisition module of the first-level node;

[0032] A sending module is used to send the first data and the second data to a database through the first database channel.

[0033] In a fifth aspect, an embodiment of the present invention provides a computer device, comprising: a processor and a memory, wherein the processor is used to execute a data processing program stored in the memory to implement the data processing method described in any one of the second aspect or the third aspect.

[0034] In the sixth aspect, an embodiment of the present invention provides a storage medium, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method described in any one of the second aspect or the third aspect above.

[0035] The data processing system provided by the embodiment of the present invention includes: a primary node, a secondary node, and a database; the secondary node is used to send the second data to the database according to the operating status of the primary node; the primary node is used to receive the second data sent by the secondary node when the operating status is normal, and send the first data and the second data in the primary node to the database; the database is used to receive the first data and the second data sent by the primary node when the operating status is normal, or, when the operating status is abnormal, receive the second data sent by the secondary node. Thus, the way to write the second data in the secondary node (slave node) and the first data in the primary node into the database can be determined according to the current operating status of the primary node (master node), ensuring that the data in each normally operating node can be written into the database, and when the primary node is unavailable, it will not affect the writing of the data of the secondary node into the database, thereby improving the efficiency of writing the data of the node into the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of another data processing system provided by an embodiment of the present invention;

[0038] Figure 3 A flowchart of a data processing method provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a flow chart of another data processing method provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the structure of another data processing device provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0045] Figure 1 A schematic diagram of a data processing system provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the system specifically includes:

[0046] Primary node 101, secondary node 102, database 103;

[0047] The secondary node 102 is used to send second data to the database according to the operating status of the primary node;

[0048] The primary node 101 is configured to receive the second data sent by the secondary node when the operation status is normal operation, and send the first data and the second data in the primary node to the database;

[0049] The database 103 is used to receive the first data and the second data sent by the primary node when the operating status is normal operation, or to receive the second data sent by the secondary node when the operating status is abnormal operation.

[0050] The data processing system provided in the embodiment of the present invention may be a distributed system (for example, a multi-node distributed data collection and reporting system), wherein the primary node is a master node and the secondary node is a slave node, and each logical unit of the distributed system includes a primary node and multiple secondary nodes. The secondary node may obtain the operating status of the current primary node, and the operating status includes: normal operation, indicating that the primary node is currently available, and abnormal operation, indicating that the primary node is unavailable.

[0051] When the primary node operates normally, the secondary node can send the second data in the secondary node to the primary node, and the primary node sends the second data and the first data in the primary node to the database. When the primary node operates abnormally, the secondary node can send the second data to the database.

[0052] In a possible implementation, the primary node includes: a first database channel, and the secondary node includes: a second reporting channel;

[0053] The second reporting channel is used to send the second data to the first database channel when the operating status is normal operation;

[0054] The first database channel is used to receive the second data and send the first data and the second data to the database when the operating status is normal operation.

[0055] In this embodiment, two channels are pre-set in each node, namely: a reporting channel (report_chan) and a database channel (mongo_chan). The two channels can be memory-level queues, so both channels can cache data when receiving data. Therefore, the first-level node contains: a first reporting channel and a first database channel, and the second-level node contains: a second reporting channel and a second database channel. When the first-level node operates normally, the second-level node will write the collected second data into the second reporting channel, and report the second data to the first-level node through the second reporting channel. The first-level node will write the first data collected at this node and the second data received into the first database channel, and then the first-level node will consume the first data and the second data in its first database channel and write them into the database.

[0056] In a possible implementation manner, the secondary node further includes: a second database channel;

[0057] The second database channel is used to obtain the second data in the second reporting channel and send the second data to the database when the operating status is abnormal operation and the new first-level node has not been determined currently.

[0058] In this embodiment, when the first-level node operates abnormally, it means that the current first-level node is unavailable. At this time, it is determined whether there is a new first-level node. If not, the second-level node writes the second data in the second reporting channel into the second database channel. The second-level node will consume the second data in its second database channel and write it into the database.

[0059] In a possible implementation, the system further includes: a target service, configured to determine a primary node and a plurality of secondary nodes from a plurality of nodes in the distributed system;

[0060] The target service is further used to: when the primary node runs abnormally, select a target secondary node from the plurality of secondary nodes as a new primary node;

[0061] The first-level node also includes: a first reporting channel, which is used to send the first data of the first-level node to the new first database channel of the new first-level node when the first-level node recovers from abnormal operation to normal operation and the new first-level node has been determined.

[0062] In this embodiment, the target service may be a redis service in a distributed system. Before determining the primary node and the secondary node, a master node is selected from a distributed logical unit as the primary node and a slave node is selected as the secondary node through the master selection mechanism of the redis service. When the primary node runs abnormally, the target service may also select a target secondary node from multiple secondary nodes as a new primary node through the master selection mechanism.

[0063] The currently unavailable first-level node includes the first reporting channel. When the operation status of the first-level node is restored to normal operation, it is determined whether a new first-level node has been determined. If not, the second-level node continues to report the second data to the first-level node through the second reporting channel. The first-level node will write the first data collected at this node and the second data received into the first database channel to report to the database through the first database channel. If so, it is determined that the original first-level node is downgraded to a second-level node, that is, the same steps as the second-level node are performed. The original first-level node will write the collected first data into the first reporting channel and report the first data to the new first-level node through the first reporting channel. The new first-level node will write the new first data collected at this node and the first data received into the new first database channel, and then the new first-level node will consume the new first data and the first data in its new first database channel and write them into the database. At the same time, the second-level node will also report the second data to the new first database channel of the new first-level node to report to the database through the new first channel.

[0064] As an example, Figure 2 FIG. 1 is a schematic diagram of the structure of another data processing system provided by an embodiment of the present invention. Figure 2 As shown, the system specifically includes:

[0065] Redis service, primary node, secondary node, and database (mongoDB database), both primary node and secondary node include: data acquisition module (for example, data acquisition is performed through shell script and python script) and data reporting module, and two channels are set in each data acquisition module, namely database channel (mongo_chan) and reporting channel (report_chan). In this embodiment, the primary node in the distributed system serves as the master node, responsible for consuming the data in all mongo_chan channels and writing to the database. The remaining nodes are secondary nodes, which report the data to the primary node through report_chan. When the primary node fails, the secondary node will automatically switch to the degraded mode, temporarily store the data in mongo_chan, and initialize the database connection by itself to achieve smooth degraded writing of data.

[0066] Implementation mechanism: Based on Redis's master election mechanism, the node regularly checks the health status and renewal status of the master node to ensure that there is only one primary node in the distributed system and the rest are secondary nodes.

[0067] Role allocation: When a node is selected as a primary node, it initializes the MongoDB database connection pool and is responsible for the final writing of all data; the secondary node does not initialize the MongoDB connection, but sends the data to the primary node through the reporting mechanism.

[0068] Data collection method: Each data collection module periodically collects system resource data and generates a data packet to obtain first data and second data. The data type is map[string]interface{}.

[0069] Channel allocation logic: The node role is determined by the master selection module, and different channels are written according to the node role:

[0070] Level 1 node: The first data collected is directly written into mongo_chan, and the data reporting module consumes data from mongo_chan and writes it into MongoDB.

[0071] Secondary node: The collected secondary data is written to report_chan and reported to mongo_chan of the primary node through a Web request.

[0072] The data reporting module, as a Web service, receives and consumes data in mongo_chan and report_chan to realize data flow.

[0073] Normal operation mode of the primary node: the primary node consumes data from mongo_chan and writes it to MongoDB, and the secondary node consumes data from report_chan and reports it to the primary node.

[0074] Abnormal operation mode of the primary node: When the primary node is unavailable, the secondary node initializes the MongoDB connection, switches the data in report_chan to mongo_chan and writes it to the database by itself.

[0075] Recovery mechanism: After the master election is completed, the new primary node begins to take on the writing task of mongo_chan, the secondary node closes the temporary database connection, and resumes the normal reporting process.

[0076] The system also includes a listening module (listen), which listens to the reporting request from the secondary node in the primary node, writes the secondary node data into mongo_chan and finally writes it into MongoDB.

[0077] After the primary node fails and recovers, ensure that the system re-receives the second data from all secondary nodes to maintain data consistency.

[0078] The data processing system provided by the embodiment of the present invention includes: a primary node, a secondary node, and a database; the secondary node is used to send the second data to the database according to the running state of the primary node; the primary node is used to receive the second data sent by the secondary node when the running state is normal, and send the first data and the second data in the primary node to the database; the database is used to receive the first data and the second data sent by the primary node when the running state is normal, or, when the running state is abnormal, receive the second data sent by the secondary node. Thus, the way to write the second data in the secondary node (slave node) and the first data in the primary node into the database can be determined according to the current running state of the primary node (master node), ensuring that the data in each normally running node can be written into the database, and when the primary node is unavailable, it will not affect the writing of the data of the secondary node into the database, thereby improving the efficiency of writing the data of the node into the database.

[0079] Figure 3 A flow chart of a data processing method provided by an embodiment of the present invention is as follows: Figure 3 As shown, the method specifically includes:

[0080] S11. Acquire second data sent by a second reporting channel of the secondary node through a first database channel in the primary node.

[0081] The data processing method provided by the embodiment of the present invention is applied to a primary node in a distributed system, and the distributed system also includes multiple secondary nodes and a database. The primary node can be a master node selected from a logical unit in the distributed system through the master selection mechanism of the redis service, and the secondary node is a slave node in the logical unit. The primary node includes a first database channel and a first reporting channel, and the secondary node includes a second database channel and a second reporting channel. The primary node initializes the database connection pool and establishes a communication connection between the first database channel and the database.

[0082] When the primary node is in normal operation, the data acquisition module of the secondary node periodically collects the second data from the distributed system, writes it into the second reporting channel, and reports the second data to the first database channel of the primary node through the second reporting channel. The first database channel temporarily stores the second data after receiving it.

[0083] S12, acquiring first data from the data acquisition module of the primary node;

[0084] S13. Send the first data and the second data to a database through the first database channel.

[0085] In this embodiment, the data acquisition module of the primary node periodically obtains the first data from the distributed system and writes the first data into the first database channel. At this time, the first database channel includes: the first data and the second data. The primary node consumes the first data and the second data of the first data channel in the writing order and reports them to the database.

[0086] When the operation status of the first-level node is abnormal, the first database channel is disconnected from the database. When the first-level node recovers to normal operation, it is determined whether there is a new first-level node. If not, the first database channel is reconnected to the database and steps S11-S13 are repeated. If so, the first-level node is used as a second-level node, and the first data (including: historically stored and newly acquired first data) and the second data (historically stored second data) in the first database channel are sent to the first reporting channel, so as to report the first data and the second data to the new first database channel of the new first-level node through the first reporting channel, and write the first data and the second data into the database through the new first database channel.

[0087] The data processing method provided by the embodiment of the present invention obtains the second data sent by the second reporting channel of the secondary node through the first database channel in the primary node; obtains the first data from the data acquisition module of the primary node; and sends the first data and the second data to the database through the first database channel. In this way, when the primary node is operating normally, the secondary node does not directly connect to the database, but only obtains data and writes it to the database through the first database channel of the primary node, which significantly reduces the database connection pressure and improves the performance and scalability of the system. At the same time, each node uses the same infrastructure, does not distinguish between data processing logic, and performs specific operations according to different node roles, which simplifies system operation and maintenance. After the primary node failure is recovered, the data can be written to the database through the new primary node to achieve efficient system recovery.

[0088] Figure 4 A flow chart of another data processing method provided by an embodiment of the present invention is as follows: Figure 4 As shown, the method specifically includes:

[0089] S21. Determine whether the primary node operates normally.

[0090] The data processing method provided by the embodiment of the present invention is applied to a secondary node in a distributed system. The distributed system also includes a primary node and a database. The primary node can be a master node selected from a logical unit in the distributed system through the master selection mechanism of the redis service, and the secondary node is a slave node in the logical unit. The primary node includes a first database channel and a first reporting channel, and the secondary node includes a second database channel and a second reporting channel. The primary node initializes the database connection pool and establishes a communication connection between the first database channel and the database. After the connection, the first database channel of the primary node can obtain the second data reported by the second reporting channel, and write the second data and the first data in the primary node into the database through the first database channel.

[0091] Before sending data to the primary node, the secondary node determines whether the current operating status of the primary node is normal.

[0092] S22. When the primary node operates normally, the second data in the secondary node is sent to the first database channel of the primary node through the second reporting channel of the secondary node, so that the first database channel sends the second data and the first data in the primary node to the database.

[0093] In this embodiment, when the judgment result is that the first-level node is operating normally, the second-level node will write the collected second data into the second reporting channel, and report the second data to the first-level node through the second reporting channel. The first-level node will write the first data collected at this node and the second data received into the first database channel, and then the first-level node will consume the first data and the second data in its first database channel and write them into the database.

[0094] S23. When the primary node operates abnormally and a new primary node has not been determined currently, the second data in the second reporting channel is sent to the second database channel of the secondary node, so as to send the second data to the database through the second database channel.

[0095] In this embodiment, when the primary node runs abnormally, it means that the current primary node is unavailable. The redis service can also select a target secondary node from multiple secondary nodes as a new primary node through the master selection mechanism. At this time, it is determined whether there is a new primary node. If not, the secondary node initializes the database connection pool and establishes a communication connection between the second database channel and the database. The second data in the second reporting channel is written into the second database channel, and the secondary node consumes the second data in its second database channel and writes it into the database.

[0096] If there is a new first-level node, the second reporting channel reports the second data to the new first database channel of the new first-level node.

[0097] The data processing method provided by the embodiment of the present invention determines whether the primary node is operating normally. When the primary node is operating normally, the second data in the secondary node is sent to the first database channel of the primary node through the second reporting channel of the secondary node, so that the first database channel sends the second data and the first data in the primary node to the database. When the primary node is operating abnormally and a new primary node has not been determined at present, the second data in the second reporting channel is sent to the second database channel of the secondary node, so that the second data is sent to the database through the second database channel. In this way, it is possible to select a method for writing the second data to the database according to the operating status of the primary node, and the second data can be successfully written to the database regardless of whether the primary node is operating abnormally, thereby ensuring the success rate of data writing to the database, and when the primary node is abnormal, there is no need to wait for the primary node to return to normal, and the second data can be directly written to the database through the secondary node, thereby improving the efficiency of data writing to the database.

[0098] Figure 5 A structural diagram of a data processing device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the device specifically includes:

[0099] A first acquisition module 51 is used to acquire the second data sent by the second reporting channel of the secondary node through the first database channel in the primary node;

[0100] A second acquisition module 52, configured to acquire first data from the data acquisition module of the primary node;

[0101] The first sending module 53 is used to send the first data and the second data to the database through the first database channel.

[0102] In a possible implementation, the first determination module 54 is configured to determine whether a new first-level node has been determined when the operation status of the first-level node is switched from abnormal operation to normal operation;

[0103] The first sending module is further configured to send the first data of the first-level node to a new first database channel of the new first-level node if the new first-level node has been determined currently;

[0104] The first acquisition module is further used to execute the step of acquiring the second data sent by the second reporting channel of the secondary node through the first database channel in the primary node if the new primary node is not determined currently.

[0105] The data processing device provided in this embodiment may be as follows Figure 5 The device shown in FIG. 1 can perform the following steps: Figure 3 All steps of the data processing method in Figure 3 For details on the technical effects of the data processing method shown, please refer to Figure 3 For the sake of brevity, the relevant description is not repeated here.

[0106] Figure 6 A structural diagram of another data processing device provided by an embodiment of the present invention is as follows: Figure 6 As shown, the device specifically includes:

[0107] The second judgment module 61 is used to judge whether the primary node is running normally;

[0108] A second sending module 62 is used for sending the second data in the secondary node to the first database channel of the primary node through the second reporting channel of the secondary node when the primary node operates normally, so that the first database channel sends the second data and the first data in the primary node to the database;

[0109] The third sending module 63 is used to send the second data in the second reporting channel to the second database channel of the secondary node when the primary node operates abnormally and a new primary node has not been determined yet, so as to send the second data to the database through the second database channel.

[0110] The data processing device provided in this embodiment may be as follows Figure 6 The device shown in FIG. 1 can perform the following steps: Figure 4 All steps of the data processing method in Figure 4 For details on the technical effects of the data processing method shown, please refer to Figure 4 For the sake of brevity, the relevant description is not repeated here.

[0111] Figure 7 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention, Figure 7 The computer device 700 shown includes: at least one processor 701, a memory 702, at least one network interface 704 and other user interfaces 703. The various components in the computer device 700 are coupled together via a bus system 705. It is understood that the bus system 705 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 705 is not described in detail. Figure 7 Various buses are labeled as bus system 705 .

[0112] The user interface 703 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0113] It can be understood that the memory 702 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 702 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0114] In some implementations, the memory 702 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 7021 and an application 7022 .

[0115] The operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 7022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 7022.

[0116] In the embodiment of the present invention, by calling the program or instruction stored in the memory 702, specifically, the program or instruction stored in the application 7022, the processor 701 is used to execute the method steps provided by each method embodiment, for example, including:

[0117] Applied to a primary node in a distributed system, the distributed system also including a plurality of secondary nodes, the method comprising:

[0118] Acquire, through the first database channel in the first-level node, second data sent by the second reporting channel of the second-level node;

[0119] Acquire first data from a data acquisition module of the primary node;

[0120] The first data and the second data are sent to a database through the first database channel.

[0121] Applied to a secondary node in a distributed system, the distributed system also including a primary node, the method comprising:

[0122] Determine whether the primary node is operating normally;

[0123] When the primary node operates normally, the second data in the secondary node is sent to the first database channel of the primary node through the second reporting channel of the secondary node, so that the first database channel sends the second data and the first data in the primary node to the database;

[0124] When the primary node operates abnormally and a new primary node has not been determined currently, the second data in the second reporting channel is sent to the second database channel of the secondary node, so as to send the second data to the database through the second database channel.

[0125] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the above method in combination with its hardware.

[0126] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0127] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0128] The computer device provided in this embodiment may be Figure 7 The computer device shown in , can execute Figure 3-4 All steps of the data processing method in Figure 3-4 For details on the technical effects of the data processing method shown, please refer to Figure 3-4 For the sake of brevity, the relevant description is not repeated here.

[0129] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0130] When one or more programs in the storage medium can be executed by one or more processors, the data processing method executed on the device side can be implemented.

[0131] The processor is used to execute the data processing program stored in the memory to implement the following steps of the data processing method performed on the device side:

[0132] Applied to a primary node in a distributed system, the distributed system also including a plurality of secondary nodes, the method comprising:

[0133] Acquire, through the first database channel in the first-level node, second data sent by the second reporting channel of the second-level node;

[0134] Acquire first data from a data acquisition module of the primary node;

[0135] The first data and the second data are sent to a database through the first database channel.

[0136] Applied to a secondary node in a distributed system, the distributed system also including a primary node, the method comprising:

[0137] Determine whether the primary node is operating normally;

[0138] When the primary node operates normally, the second data in the secondary node is sent to the first database channel of the primary node through the second reporting channel of the secondary node, so that the first database channel sends the second data and the first data in the primary node to the database;

[0139] When the primary node operates abnormally and a new primary node has not been determined currently, the second data in the second reporting channel is sent to the second database channel of the secondary node, so as to send the second data to the database through the second database channel.

[0140] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0141] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0142] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing system, characterized in that: include: Primary node, secondary node, database; The secondary node is used to send second data to the database according to the operating status of the primary node; The primary node is configured to receive the second data sent by the secondary node when the operation status is normal operation, and send the first data and the second data in the primary node to the database; The database is used to receive the first data and the second data sent by the primary node when the operating status is normal operation, or to receive the second data sent by the secondary node when the operating status is abnormal operation.

2. The system according to claim 1, characterized in that The first-level node includes: a first database channel, and the second-level node includes: a second reporting channel; The second reporting channel is used to send the second data to the first database channel when the operating status is normal operation; The first database channel is used to receive the second data and send the first data and the second data to the database when the operating status is normal operation.

3. The system according to claim 2, characterized in that The secondary node also includes: a second database channel; The second database channel is used to obtain the second data in the second reporting channel and send the second data to the database when the operating status is abnormal operation and the new first-level node has not been determined currently.

4. The system according to claim 3, characterized in that The system further includes: a target service for determining a primary node and a plurality of secondary nodes from a plurality of nodes in the distributed system; The target service is further used to: when the primary node runs abnormally, select a target secondary node from the plurality of secondary nodes as a new primary node; The first-level node also includes: a first reporting channel, which is used to send the first data of the first-level node to the new first database channel of the new first-level node when the first-level node recovers from abnormal operation to normal operation and the new first-level node has been determined.

5. A data processing method, characterized in that: Applied to a primary node in a distributed system, the distributed system also including a plurality of secondary nodes, the method comprising: Acquire, through the first database channel in the first-level node, second data sent by the second reporting channel of the second-level node; Acquire first data from a data acquisition module of the primary node; The first data and the second data are sent to a database through the first database channel.

6. The method according to claim 5, characterized in that The method further comprises: When the operation state of the first-level node switches from abnormal operation to normal operation, determining whether a new first-level node has been determined; If the new first-level node has been determined, the first data of the first-level node is sent to a new first database channel of the new first-level node; If the new primary node is not currently determined, the step of acquiring the second data sent by the second reporting channel of the secondary node through the first database channel in the primary node is performed.

7. A data processing method, characterized in that: Applied to a secondary node in a distributed system, the distributed system also including a primary node, the method comprising: Determine whether the primary node is operating normally; When the primary node operates normally, the second data in the secondary node is sent to the first database channel of the primary node through the second reporting channel of the secondary node, so that the first database channel sends the second data and the first data in the primary node to the database; When the primary node operates abnormally and a new primary node has not been determined currently, the second data in the second reporting channel is sent to the second database channel of the secondary node, so as to send the second data to the database through the second database channel.

8. A data processing device, characterized in that: Applied to a primary node in a distributed system, the distributed system also includes a plurality of secondary nodes, the device comprising: A first acquisition module, configured to acquire, through the first database channel in the first-level node, second data sent by the second reporting channel of the second-level node; A second acquisition module, used to acquire first data from the data acquisition module of the first-level node; The first sending module is used to send the first data and the second data to the database through the first database channel.

9. A computer device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 5 or 6.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method according to any one of claims 5 or 6.

Images