Data processing method and apparatus

Abstract

The application discloses a data processing method and device, and relates to the technical field of computers. A specific implementation manner of the method comprises the following steps: determining whether a first type of task execution cycle is reached; if yes, generating an election instruction corresponding to the first type of task, and performing election according to the election instruction; and if elected as a leader node corresponding to the first type of task, executing the first type of task. The implementation manner splits the main function of the system into the first type of task and a second type of task, executes the first type of task through dynamic election, realizes a decentralized architecture, and does not have a central node in the cluster, so that the failure of any node can be taken over by other nodes in the cluster, and the size of the cluster can be dynamically horizontally expanded. On the other hand, all the nodes in the cluster actively acquire the second type of task, replace the mode of being forced to accept tasks in a mainstream architecture, and realize automatic balanced distribution of tasks according to the processing capacity of the nodes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method and device. BACKGROUND

[0002] With the rise of data platform, the business carried by enterprise data center is more and more complex and diverse, and the exchange of internal data of enterprises is more and more frequent. How to efficiently and safely make data converge and process to empower business is an important indicator of data platform construction. At present, in the real-time data exchange system, a server+agent distributed cluster is usually used for data exchange. Among them, the server is responsible for discovering real-time data files and issuing them to the corresponding agent. The agent is responsible for reading the file from the source and transmitting it to the data platform after receiving the task. In this architecture, the server node is the center node of the whole cluster and undertakes all the computing tasks in the cluster. Moreover, since the cluster can only have one center node, the upper limit of the cluster capacity is limited by the processing capacity of the server. In addition, in terms of real-time data exchange, the open source datax is a relatively mainstream product. This product separates data reading and writing, and the reading module divides the source data into several parallel tasks through configuration, reads the data into the cache, and the writing module reads the data from the cache and writes it to the target. However, datax only solves the problem of how to transmit data from the source to the target, and does not provide data conversion and trimming in the transmission process. Therefore, it only solves the E(Extract) and L(Load) in ETL(Extract-Transform-Load), but the T(Transform) is missing. Among them, ETL is used to describe the process of extracting data from the source, transforming and loading it to the target. SUMMARY

[0003] Therefore, the embodiments of the present application provide a data processing method and device, which abstract the task type, split the main functions of the system into first type tasks and second type tasks, execute the first type tasks by dynamic master selection, realize a decentralized architecture, there is no center node in the cluster, and the failure of any node can be taken over by other nodes in the cluster, solve the problem of center node limiting the size of the cluster, the size of the cluster can be dynamically horizontally expanded, and there is no upper limit. On the other hand, all nodes in the cluster will actively obtain the second type tasks, replacing the forced task acceptance mode in the mainstream architecture, and realizing automatic balanced task allocation according to the processing capacity of the nodes.

[0004] To achieve the above object, according to an aspect of an embodiment of the present application, a data processing method is provided, which is applied to a node in a cluster; the method comprises: determining whether a pre-configured execution cycle of a first type task is reached; if yes, generating an election instruction corresponding to the first type task, and performing election according to the election instruction; and in the case of being elected as a leader node corresponding to the first type task, executing the first type task.

[0005] Optionally, before determining whether the pre-configured execution cycle of the first type task is reached, the method further comprises: executing a pre-configured second type task.

[0006] Optionally, the first type task comprises a data discovery task.

[0007] In the case of being elected as the leader node corresponding to the first type task, executing the first type task comprises: in the case of being elected as a first leader node corresponding to the data discovery task, executing the data discovery task to determine whether there is new data; and if there is new data, writing summary information of the new data into a preset cache queue.

[0008] Executing the pre-configured second type task comprises: according to the summary information, downloading the new data from a data production end to a local end, and according to a preset processing rule, processing the new data, and uploading the processed new data to a transfer directory.

[0009] Optionally, the first type task further comprises a data clock task.

[0010] In the case of being elected as the leader node corresponding to the first type task, executing the first type task comprises: in the case of being elected as a second leader node corresponding to the data clock task, executing the data clock task, and periodically converting the new data in the transfer directory according to a preset rule.

[0011] Optionally, the first type task further comprises a data warehousing task.

[0012] In the case of being elected as the leader node corresponding to the first type task, executing the first type task comprises: in the case of being elected as a third leader node corresponding to the data warehousing task, executing the data warehousing task to write the new data periodically converted in the transfer directory into a database.

[0013] Optionally, writing the new data periodically converted in the transfer directory into the database comprises: according to a period of the converted new data, determining a data table corresponding to the new data, and writing the new data into the corresponding data table.

[0014] Optionally, the method determines whether there is new data according to a process that compares a file list in a to-be-collected catalog with a downloaded file list to determine whether there is new data.

[0015] Optionally, according to the summary information, the new data is downloaded from a data production end to a local, and the new data is processed according to a preset processing rule, and the processed new data is uploaded to a transfer catalog.

[0016] According to the summary information, the new data is downloaded from a data production end to a local, and the new data is processed according to a preset processing rule, and the processed new data is uploaded to a transfer catalog.

[0017] Optionally, processing the new data according to the preset processing rule includes: obtaining a target processing rule corresponding to the new data from a preset field-level processing function library, and processing the new data according to the target processing rule.

[0018] To achieve the above object, according to another aspect of the embodiment of the present application, a data processing method is provided, comprising: configuring a first type task, and setting an execution period of the first type task; in the case of reaching the execution period of the first type task, generating an election instruction corresponding to the first type task, triggering nodes in a cluster to perform election based on the election instruction, and determining a leader node corresponding to the first type task, so that the leader node executes the first type task.

[0019] Optionally, the first type task includes a data discovery task.

[0020] Determining a leader node corresponding to the first type task, so that the leader node executes the first type task, includes:

[0021] Determining a first leader node corresponding to the data discovery task, so that the first leader node determines whether there is new data; if there is new data, writing summary information of the new data into a preset cache queue.

[0022] The process that the nodes in the cluster execute the second type task includes:

[0023] acquire the summary information from the preset cache queue, download the new data from a data production end to a local according to the summary information, and process the new data according to a preset processing rule, and upload the processed new data to a transfer directory.

[0024] Optionally, the summary information comprises a modification time of the new data; and the first type task further comprises a data clocking task.

[0025] determining a leader node corresponding to the first type task, so that the leader node executes the first type task.

[0026] determining a second leader node corresponding to the data discovery task, so that the second leader node performs periodic conversion on the new data in the transfer directory according to a preset rule.

[0027] Optionally, the first type task further comprises a data warehousing task.

[0028] determining a third leader node corresponding to the data discovery task, so that the third leader node writes the new data after periodic conversion in the transfer directory into a database.

[0029] Optionally, writing the new data after periodic conversion in the transfer directory into a database comprises: determining a data table corresponding to the new data according to a period after conversion of the new data, and writing the new data into the corresponding data table.

[0030] Optionally, downloading the new data from a data production end to a local according to the summary information, processing the new data according to a preset processing rule, and uploading the processed new data to a transfer directory comprises:

[0031] reading the new data from a first ring array in the local after downloading the new data from a data production end to the local according to the summary information, processing the new data, writing the processed new data into a second ring array in the local, reading the processed new data from the second ring array, and uploading the processed new data to a transfer directory.

[0032] Optionally, processing the new data according to a preset processing rule comprises: acquiring a target processing rule corresponding to the new data from a preset field-level processing function library, and processing the new data according to the target processing rule.

[0033] To achieve the above object, according to another aspect of the embodiments of the present application, a data processing apparatus is provided, comprising:

[0034] a determining module configured to determine whether a preconfigured execution cycle of a first type task is reached;

[0035] an election module configured to generate an election instruction corresponding to the first type task, and perform election according to the election instruction;

[0036] a task execution module configured to execute the first type task if elected as a leader node corresponding to the first type task.

[0037] Optionally, the task execution module is further configured to execute a preconfigured second type task.

[0038] Optionally, the first type task comprises a data discovery task.

[0039] The task execution module is further configured to execute the data discovery task to determine whether there is new data, write summary information of the new data into a preconfigured cache queue if there is new data, download the new data from a data production end to a local according to the summary information, and process the new data according to a preconfigured processing rule, and upload the processed new data to a transfer directory.

[0040] Optionally, the first type task further comprises a data conversion task.

[0041] The task execution module is further configured to execute the data conversion task to periodically convert the new data in the transfer directory according to a preconfigured rule.

[0042] Optionally, the first type task further comprises a data warehousing task.

[0043] The task execution module is further configured to execute the data warehousing task to write the new data periodically converted in the transfer directory into a database.

[0044] Optionally, the task execution module is further configured to determine a data table corresponding to the new data according to a period of the converted new data, and write the new data into the corresponding data table.

[0045] Optionally, the task execution module is further configured to compare a file list in a to-be-collected directory with a downloaded file list to determine whether there is new data.

[0046] Optionally, the task execution module is further configured to: download the new data from the data production end to a first local ring array according to the summary information; read the new data from the first ring array for processing; write the processed new data into a second local ring array; read the processed new data from the second ring array; and upload the processed new data to a transfer directory.

[0047] Optionally, the task execution module is further configured to: obtain a target processing rule corresponding to the new data from a preset field-level processing function library; and process the new data according to the target processing rule.

[0048] To achieve the above object, according to a further aspect of the embodiments of the present application, an electronic device is provided, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method of the embodiments of the present application.

[0049] To achieve the above object, according to a further aspect of the embodiments of the present application, a computer readable medium is provided, which stores a computer program, when the program is executed by a processor, the data processing method of the embodiments of the present application is implemented.

[0050] The above embodiment has the following advantages or beneficial effects: by abstracting the task type, the main functions of the system are divided into the first type of task and the second type of task, the first type of task is executed by dynamic master selection, a decentralized architecture is realized, there is no central node in the cluster, and the failure of any node can be taken over by other nodes in the cluster, solving the problem of central node limiting the size of the cluster, the size of the cluster can be dynamically horizontally expanded, and there is no upper limit; on the other hand, all nodes in the cluster will actively obtain the second type of task, replacing the forced task acceptance mode in the mainstream architecture, and realizing automatic balanced distribution of tasks according to the processing capacity of the nodes.

[0051] The further effects of the above non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings are used to better understand the present application, and do not constitute an improper limitation on the present application. Among them:

[0053] Figure 1 is a schematic diagram of the main process of the data processing method of the embodiments of the present application;

[0054] Figure 2 is a schematic diagram of the main process of the data processing method of another embodiment of the present application;

[0055] Figure 3 is a schematic diagram of a sub-process of a data processing method according to another embodiment of the present application;

[0056] Figure 4 is a schematic diagram of a process of a data processing method according to another embodiment of the present application;

[0057] Figure 5 is a schematic diagram of main modules of a data processing apparatus according to an embodiment of the present application;

[0058] Figure 6 is an exemplary system architecture diagram in which an embodiment of the present application can be applied;

[0059] Figure 7 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0060] The exemplary embodiments of the present application will be described below with reference to the accompanying drawings, in which various details of the embodiments of the present application are set forth in order to provide a thorough understanding of the embodiments of the present application. It will be apparent to those skilled in the art that the embodiments described herein can be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure aspects of the embodiments of the present application.

[0061] Figure 1 and Figure 2 is a schematic diagram of a process of a data processing method according to an embodiment of the present application. Different from the above, Figure 1 is a schematic diagram of the method from the perspective of a node, Figure 2 is a schematic diagram of the method from the perspective of a system. As Figure 1 indicated, the method comprises:

[0062] Step S101: performing a pre-configured second type task;

[0063] Step S102: determining whether a performing period of a pre-configured first type task is reached;

[0064] Step S103: if yes, generating an election instruction corresponding to the first type task, and performing election according to the election instruction;

[0065] Step S104: in a case where the node is elected as a leader node corresponding to the first type task, performing the first type task.

[0066] As Figure 2 indicated, the method comprises:

[0067] Step S201: configuring a first type task and a second type task, and setting an execution period of the first type task;

[0068] Step S202: setting the nodes in the cluster to execute the second type task;

[0069] Step S203: generating an election instruction corresponding to the first type task when the execution period of the first type task is reached;

[0070] Step S204: triggering the nodes in the cluster to perform election based on the election instruction, determining a leader node corresponding to the first type task, and making the leader node execute the first type task.

[0071] In the embodiment, the main functions of the system are split into the first type task and the second type task. The first type task is a server type task, and the second type task is an agent type task. The server type task corresponds to the task that needs to be executed by the server role in the mainstream architecture. The agent type task corresponds to the task that needs to be executed by the agent role in the mainstream architecture. Among them, at the same time, all the nodes in the cluster can execute the second type task, and only the node elected as the leader node can execute the first type task. The leader node can execute the first type task and the second type task at the same time. The embodiment splits the main functions of the system into the first type task and the second type task through the abstraction of the task type, executes the first type task through dynamic leader election, realizes the decentralized architecture, and there is no central node in the cluster. The failure of any node can be taken over by other nodes in the cluster, solves the problem that the central node limits the size of the cluster, and the size of the cluster can be dynamically horizontally expanded, and there is no upper limit.

[0072] In order to perform election according to a certain time period, an execution period can be defined for the first type task. After the execution time point is reached, all the nodes in the cluster perform election, and the selected node performs the execution of the task. As an example, the first type task can be pre-configured, and the execution period of the first type task is set. When the execution period of the first type task is reached, an election instruction corresponding to the first type task is generated. Based on the election instruction, the nodes in the cluster perform election, a leader node corresponding to the first type task is determined, and the leader node executes the first type task.

[0073] Further, the first type of task (i.e., server type task) can include a data discovery task, a data transfer task, and a data warehousing task. Therefore, when the execution period of the data discovery task arrives, an election needs to be performed in the cluster to determine a first leader node for executing the data discovery task. When the execution period of the data transfer task arrives, an election also needs to be performed in the cluster to determine a second leader node for executing the data transfer task. When the execution period of the data warehousing task arrives, an election also needs to be performed in the cluster to determine a third leader node for executing the data warehousing task. In an optional embodiment, the first type of task further includes a scheduling trigger task, which is used to trigger a subsequent batch processing task after the completion of a period of data warehousing (the batch processing task refers to an operation process after data warehousing). Therefore, when the execution period of the scheduling trigger task arrives, an election also needs to be performed in the cluster to determine a fourth leader node for executing the scheduling trigger task. For the convenience of understanding, the various tasks are described in detail below.

[0074] For the data discovery task, if the execution period of the data discovery task arrives, a first election instruction corresponding to the data discovery task is generated; based on the first election instruction, the nodes in the cluster are triggered to perform an election to determine a first leader node. The first leader node executes the data discovery task to determine whether there is new data; if there is new data, the summary information of the new data is written into a preset cache queue. The summary information of the new data includes a file name, a file size, and a modification time. The first leader node can determine whether there is new data according to the following process: comparing the file list in the to-be-collected directory with the downloaded file list to determine whether there is new data. The downloaded file list is stored in the cache of the cluster, and the content of the list can be read by any node in the cluster.

[0075] For the data clock task, if the execution period of the data clock task is reached, a second election instruction corresponding to the data clock task is generated, based on the second election instruction, the nodes in the cluster are triggered to perform election, a second leader node is determined, the second leader node executes the data clock task, and the newly added data in the transit directory is periodically converted according to a preset rule. For example, the newly added data in the transit directory can be periodically converted according to the current time, that is, the time of the newly added data in the transit directory is converted. The newly added data in the transit directory is uploaded by other nodes, and the process is described below. In this embodiment, since the data from the source is real-time uninterrupted streaming data, the nodes also process streaming data, and the use end needs to analyze and process according to a certain period (month / day / hour, etc.), so the newly added data needs to be converted and updated for the "data period". For example: the use end needs to analyze the data by day, then the data period arriving after 2021-03-20 00:00:00 and before 2021-03-21 00:00:00 is all 20210320, that is, the time of the newly added data is between 2021-03-20 00:00:00-2021-03-21 00:00:00, then the period of the newly added data is recorded as 20210320. If the use end needs to analyze the data by month, then the data period arriving after 2021-03-01 00:00:00 and before 2021-04-01 00:00:00 is all 202103, that is, the time of the newly added data is between 2021-03-01 00:00:00-2021-04-01 00:00:00, then the period of the newly added data is recorded as 202103.

[0076] For the data entry task, when the execution period of the data entry task is reached, a third election instruction is generated, based on the third election instruction, the nodes in the cluster are triggered to perform election, a third leader node is determined, and the third leader node executes the data entry task and writes the newly added data in the transit directory after period conversion into a database. More specifically, the third leader node determines the data table corresponding to the newly added data according to the period of the converted newly added data, and writes the newly added data into the corresponding data table. Taking the above example as an example, the data with the period 20210320 is written into the data table corresponding to "20210320", and the data with the period 20210321 is written into the data table corresponding to "20210321".

[0077] For the second type of task, the nodes in the cluster can continuously and actively obtain summary information from the cache queue, download the new data from the data production end to the local according to the summary information, and process the new data according to the preset processing rule, and upload the processed new data to the transfer directory. In this embodiment, the summary information includes the file name of the new data, and each node can sequentially obtain the summary information from the cache queue and download the data from the data production end according to the file name in the summary information. Specifically, in this embodiment, a field-level processing function library is pre-constructed, and the target processing rule corresponding to the new data can be obtained from the preset field-level processing function library according to different data sources or different interfaces, and then the new data is processed according to the target processing rule. Further, as shown in Figure 3 the process of the node executing the second type of task can be abstracted into three functional modules, such as a reading module, a processing module and a writing module, and a ring buffer is used as a data cache between the three modules. The processing module in the middle provides a field-level processing function library, which can dynamically assemble processing combinations according to different processing rules of different interfaces to meet the needs of multiple interfaces with one code. Therefore, the node can download the new data from the data production end to the first ring buffer in the local according to the summary information, read the new data from the first ring buffer, process the new data, write the processed new data to the second ring buffer in the local, read the processed new data from the second ring buffer, and upload the processed new data to the transfer directory. The process of processing the new data is abstracted into reading, converting and outputting, and a field-level processing function library is provided, which realizes the combination of different sources, processing rules and output ends, has a wider range of adaptation, and is suitable for different scenarios.

[0078] The data processing method of the embodiment of the application abstracts the task type, splits the main functions of the system into the first type of task and the second type of task, realizes the purpose of decentralization through dynamic master selection (i.e. dynamic election of leader nodes), solves the problem that the cluster center node limits the cluster size, and because there is no center node in the cluster, the failure of any host can be taken over by other nodes in the cluster, so it will not affect the overall function of the cluster. Moreover, with the expansion of the business, the size of the cluster can also be dynamically horizontally expanded, and there is no upper limit. On the other hand, all nodes in the cluster will actively obtain the second type of task, which replaces the mode of being forced to accept tasks in mainstream architecture, and realizes automatic balanced allocation of tasks according to the processing capacity of the nodes.

[0079] Figure 4 is a flowchart of a data processing method according to another embodiment of the present application. Compared with the method shown in Figure 1 , Figure 2 the method shown in Figure 4 is explained from the perspective of newly added data. The method shown in Figure 4 explains a processing flow of a data file from the source to the warehouse. As shown in Figure 4 , the method comprises the following steps:

[0080] (1) A source system generates a data file and transmits the data file to a to-be-collected directory.

[0081] (2) A node in a cluster is elected as a leader node to perform data discovery. After discovering a newly added file, the leader node writes summary information of the newly added file into a cache queue.

[0082] (3) A node in the cluster (the node can be any node in the cluster) obtains the summary information from the cache queue, downloads the summary information to a local node, and processes the newly added file according to a pre-configured processing rule.

[0083] (4) The node transmits the processed newly added file to a transfer directory.

[0084] (5) A node in the cluster is elected as a leader node, which performs periodic conversion on the newly added data in the transfer directory.

[0085] (6) A node in the cluster is elected as a leader node, which writes the newly added data after the periodic conversion in the transfer directory into a database.

[0086] Figure 5 is a schematic diagram of main modules of a data processing apparatus 500 according to an embodiment of the present application. As shown in Figure 5 , the apparatus 500 comprises the following modules:

[0087] A determination module 501 is configured to determine whether a pre-configured execution period of a first type of task is reached.

[0088] An election module 502 is configured to generate an election instruction corresponding to the first type of task, and perform election according to the election instruction.

[0089] A task execution module 503 is configured to, in a case where the task execution module 503 is elected as a leader node corresponding to the first type of task, execute the first type of task.

[0090] Optionally, the task execution module is further configured to execute a pre-configured second type of task.

[0091] Optionally, the first type of task comprises a data discovery task.

[0092] The task execution module is further configured to execute the data discovery task to determine whether there is new data, write summary information of the new data into a preset cache queue if there is new data, download the new data from a data production end to a local according to the summary information, and process the new data according to a preset processing rule and upload the processed new data to a transfer directory.

[0093] Optionally, the first type of task further includes a data transfer task.

[0094] The task execution module is further configured to execute the data transfer task to periodically convert the new data in the transfer directory according to a preset rule.

[0095] Optionally, the first type of task further includes a data warehousing task.

[0096] The task execution module is further configured to execute the data warehousing task to write the new data converted periodically in the transfer directory into a database.

[0097] Optionally, the task execution module is further configured to determine a data table corresponding to the new data according to a period of the converted new data, and write the new data into the corresponding data table.

[0098] Optionally, the task execution module is further configured to compare a file list in a to-be-collected directory with a downloaded file list to determine whether there is new data.

[0099] Optionally, the task execution module is further configured to download the new data from a data production end to a first ring array in a local according to the summary information, read the new data from the first ring array for processing, write the processed new data into a second ring array in the local, read the processed new data from the second ring array, and upload the processed new data to a transfer directory.

[0100] Optionally, the task execution module is further configured to obtain a target processing rule corresponding to the new data from a preset field-level processing function library, and process the new data according to the target processing rule.

[0101] The data processing apparatus provided in the embodiments of the present application can implement a decentralized architecture by abstracting the task type, splitting the main function of the system into a first type of task and a second type of task, and executing the first type of task by dynamic master selection, so that there is no central node in the cluster, and the failure of any node can be taken over by other nodes in the cluster, thereby solving the problem of the central node limiting the size of the cluster, and the size of the cluster can be dynamically horizontally expanded without upper limit; on the other hand, all nodes in the cluster will actively acquire the second type of task, replacing the mode of being forced to accept tasks in the mainstream architecture, and automatically balancing the distribution of tasks according to the processing capacity of the nodes.

[0102] The apparatus described above can perform the method provided in the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the embodiments can be referred to the method provided in the embodiments of the present application.

[0103] Figure 6 An exemplary system architecture 600 to which the data processing method or data processing apparatus of the embodiments of the present application can be applied is shown.

[0104] As shown in Figure 6 The system architecture 600 can include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 is used to provide a communication link medium between the terminal devices 601, 602, 603 and the server 605. The network 604 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0105] A user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0106] The terminal devices 601, 602, 603 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0107] The server 605 can be a server providing various services, such as a background management server supporting a shopping website browsed by a user using the terminal devices 601, 602, 603. The background management server can analyze and process received product information query requests and other data, and feed back the processing results (such as target push information, product information) to the terminal devices.

[0108] It should be noted that the data processing method provided by the embodiments of the present application is generally executed by the server 605, and accordingly, the data processing apparatus is generally arranged in the server 605.

[0109] It should be understood that, Figure 6 The number of terminal devices, networks and servers in the system is merely illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs.

[0110] Reference is made below to Figure 7 which shows a structural schematic diagram of a computer system 700 of a terminal device suitable for use to implement the embodiments of the present application. Figure 7 The terminal device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0111] As shown in Figure 7 , the computer system 700 includes a central processing unit (CPU) 701 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or programs loaded from a storage portion 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The CPU 701, the ROM 702 and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0112] The following components are connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable recording medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0113] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program in accordance with the embodiments disclosed herein. For example, embodiments disclosed herein include a computer program product which includes a computer program tangibly embodied on a computer readable medium, the computer program including program code for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 709 and / or installed from a removable media 711. When the computer program is executed by the central processing unit (CPU) 701, the above-described functions defined in the system of the present application are performed.

[0114] It should be noted that the computer readable medium shown in the present application can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium that can send, propagate or transmit the program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF or the like, or any suitable combination thereof.

[0115] The computer program product of the present application can be a computer program embodied on a non-transitory computer readable medium. When the computer program runs on a computer, the computer is enabled to perform the functions of the embodiments of the present application.

[0116] The modules described in the embodiments of the present application can be implemented by software or hardware. The modules described can be arranged in a processor, for example, a processor can be described as including a sending module, an obtaining module, a determining module and a first processing module. In some cases, the names of the modules do not constitute a limitation on the modules themselves, for example, the sending module can also be described as a "module for sending a picture obtaining request to a connected server".

[0117] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist separately without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0118] determining whether an execution period of a preconfigured first type task is reached;

[0119] if yes, generating an election instruction corresponding to the first type task, and performing election according to the election instruction;

[0120] in a case of being elected as a leader node corresponding to the first type task, executing the first type task.

[0121] The technical scheme of the embodiment of the present application abstracts the task type, splits the main function of the system into a first type task and a second type task, executes the first type task by dynamic master selection, realizes a decentralized architecture, there is no central node in the cluster, and the failure of any node can be taken over by other nodes in the cluster, thereby solving the problem of the central node limiting the cluster size, the size of the cluster can be dynamically horizontally expanded, and there is no upper limit; on the other hand, all nodes in the cluster will actively acquire the second type task, replacing the mode of being forced to accept tasks in the mainstream architecture, and realizing automatic balanced distribution of tasks according to the processing capacity of the nodes.

[0122] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method is applied to nodes in a cluster, and there is no central node in the cluster; the method comprises: determining whether a pre-configured execution period of a first type task is reached; if yes, generating an election instruction corresponding to the first type task, and performing election according to the election instruction; in the case of being elected as a leader node corresponding to the first type task, executing the first type task; wherein the first type task comprises one or more of a data discovery task, a data transfer task, a data warehousing task, and a scheduling trigger task, and a leader node for executing different first type tasks is determined; a node in the cluster actively acquires a second type task.

2. The method of claim 1, wherein, Before determining whether the pre-configured execution period of the first type task is reached, the method further comprises: executing a pre-configured second type task.

3. The method of claim 2, wherein, The first type task comprises a data discovery task; in the case of being elected as a leader node corresponding to the first type task, executing the first type task comprises: in the case of being elected as a first leader node corresponding to the data discovery task, executing the data discovery task to determine whether there is new data; if there is new data, writing summary information of the new data into a preset cache queue; executing the pre-configured second type task comprises: downloading the new data from a data production end to a local according to the summary information, and processing the new data according to a preset processing rule, and uploading the processed new data to a transfer directory.

4. The method of claim 3, wherein, The first type task further comprises a data transfer task; in the case of being elected as a leader node corresponding to the first type task, executing the first type task comprises: in the case of being elected as a second leader node corresponding to the data transfer task, executing the data transfer task to periodically convert the new data in the transfer directory according to a preset rule.

5. The method of claim 4, wherein, The first type task further comprises a data warehousing task; in the case of being elected as a leader node corresponding to the first type task, executing the first type task comprises: in the case of being elected as a third leader node corresponding to the data warehousing task, executing the data warehousing task to write the new data periodically converted in the transfer directory into a database.

6. The method of claim 5, wherein, writing the new data periodically converted in the transfer directory into a database comprises: determining a data table corresponding to the new data according to the period of the converted new data, and writing the new data into the corresponding data table.

7. The method of claim 3, wherein, The method determines whether there is new data according to the following process: comparing a file list in a to-be-collected directory with a downloaded file list to determine whether there is new data.

8. The method of claim 3, wherein, According to the summary information, the new data is downloaded from a data production end to a local, and the new data is processed according to a preset processing rule, and the processed new data is uploaded to a transfer directory, which comprises: According to the summary information, the new data is downloaded from a data production end to a local first ring array, the new data is read from the first ring array for processing, the processed new data is written into a local second ring array, the processed new data is read from the second ring array, and the processed new data is uploaded to a transfer directory.

9. The method of claim 3, wherein, According to a preset processing rule, the new data is processed. According to the target processing rule, the new data is processed.

10. A data processing method, characterized by, It includes: Configure a first type of task and set the execution period of the first type of task; When the execution period of the first type of task is reached, an election instruction corresponding to the first type of task is generated; Based on the election instruction, trigger the nodes in the cluster to elect, determine the leader node corresponding to the first type of task, so that the leader node executes the first type of task; wherein there is no center node in the cluster; the first type of task includes one or more of data discovery task, data transfer task, data warehousing task, and scheduling trigger task; determine the leader node for executing different first type of tasks; The nodes in the cluster actively acquire a second type of task.

11. The method of claim 10, wherein, When configuring the first type of task, the method further includes configuring a second type of task and setting the nodes in the cluster to execute the second type of task.

12. The method of claim 11, wherein, The first type of task includes a data discovery task; Determine the leader node corresponding to the first type of task to make the leader node execute the first type of task, including: Determine a first leader node corresponding to the data discovery task, so that the first leader node determines whether there is new data; if there is new data, write the summary information of the new data into a preset cache queue; The process in which the nodes in the cluster execute the second type of task includes: According to the summary information, the new data is downloaded from a data production end to a local first ring array, the new data is read from the first ring array for processing, the processed new data is written into a local second ring array, the processed new data is read from the second ring array, and the processed new data is uploaded to a transfer directory.

13. The method of claim 12, wherein, The first type of task also includes a data transfer task; Determine the leader node corresponding to the first type of task to make the leader node execute the first type of task, including: Determine a second leader node corresponding to the data discovery task, so that the second leader node periodically converts the new data in the transfer directory according to a preset rule.

14. The method of claim 13, wherein, The first type of task also includes a data warehousing task; Determine the leader node corresponding to the first type of task to make the leader node execute the first type of task, including: Determine a third leader node corresponding to the data discovery task, so that the third leader node writes the new data in the transfer directory after periodic conversion into a database.

15. The method of claim 14, wherein, Writing the new data in the transfer directory after periodic conversion into a database includes: According to the period of the converted new data, a data table corresponding to the new data is determined, and the new data is written into the corresponding data table.

16. The method of claim 12, wherein, According to the summary information, the new data is downloaded from a data production end to a local, and the new data is processed according to a preset processing rule, and the processed new data is uploaded to a transfer directory. According to the summary information, the new data is downloaded from a data production end to a local, and the new data is processed according to a preset processing rule, and the processed new data is uploaded to a transfer directory.

17. The method of claim 12, wherein, According to the summary information, the new data is downloaded from a data production end to a local, and the new data is processed according to a preset processing rule, and the processed new data is uploaded to a transfer directory. According to the preset processing rule, the new data is processed, including:

18. A data processing apparatus, characterized by According to the target processing rule, the new data is processed. Including: A determination module is configured to determine whether a first type of task reaches a preset execution period; An election module is configured to generate an election instruction corresponding to the first type of task, and perform election according to the election instruction; A task execution module is configured to execute the first type of task when elected as a leader node corresponding to the first type of task; wherein, there is no central node in the cluster; the first type of task includes one or more of a data discovery task, a data transfer task, a data storage task, and a scheduling trigger task; and the leader node executes different first type of tasks; 19. An electronic device, comprising: A node in the cluster actively acquires a second type of task. Including: One or more processors; A storage device configured to store one or more programs, 20. A computer readable medium having stored thereon a computer program, characterized in that, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-17. The program is executed by the processor to implement the method of any one of claims 1-17.

Images