A data processing method and system, computer equipment, and computer storage medium

Abstract

The present invention can provide a data processing method and system, a computer device, and a computer storage medium. The data processing method may include: generating a data processing task based on a received task initiation instruction, reading the file path information of the data to be processed pointed to by the data processing task, decomposing the data processing task into multiple subtasks according to the file path information, creating a cluster sub-role for processing the subtasks, assigning at least one subtask to each cluster sub-role, and using each cluster sub-role to process at least one subtask received by each cluster sub-role, so as to collect and analyze the data to be processed. The present invention can quickly process a large amount of data through parallel distributed data collection means, and the data collection and analysis efficiency is extremely high. Based on the advantages of easy construction and expansion of clusters, it can better meet the timely processing needs of rapidly growing data, and significantly reduce the difficulty of data collection and analysis.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology. More specifically, the present invention can provide a data processing method and system, a computer device, and a computer storage medium. Background Art

[0002] With the exponential growth of data volume and the deepening of data understanding, the efficient use of data has become increasingly important, and data collection is the foundation and core of data application. There are many data collection tools on the market, such as Sqoop, Datax, Flume, and Canal, but most of these data collection tools have high requirements for data sources. The different data structures of data sources directly affect the efficiency of data collection. In actual operation, it is found that problems such as data loss or repeated data collection often occur in the existing data collection process.

[0003] Therefore, data collection using conventional technologies is difficult, and it is difficult to process rapidly growing data in a timely manner, and there is an urgent need to improve or optimize conventional technologies. Summary of the invention

[0004] In order to solve at least one problem existing in the prior art, the present invention can provide a data processing method and system, a computer device, and a computer storage medium to achieve at least one technical purpose such as improving the efficiency of data collection and analysis and reducing the difficulty of data collection and analysis.

[0005] To achieve the above technical objectives, one or more embodiments of the present invention can specifically provide a data processing method, which may include but is not limited to at least one of the following steps.

[0006] Generate a data processing task based on the received task initiation instruction.

[0007] Read the file path information of the data to be processed pointed to by the data processing task.

[0008] The data processing task is decomposed into a plurality of subtasks according to the file path information.

[0009] A cluster subrole is created to handle the subtask.

[0010] At least one subtask is assigned to each of the cluster subroles.

[0011] Each of the cluster sub-roles is utilized to process at least one sub-task received by each of them, so as to collect and analyze the data to be processed.

[0012] Further, decomposing the data processing task into a plurality of subtasks according to the file path information includes:

[0013] The upper folder path information of each file is read based on the file path information.

[0014] The data processing task is decomposed based on the upper folder path of a single file as the basis for task decomposition, and the data processing task is decomposed into multiple subtasks.

[0015] Furthermore, the data processing method also includes:

[0016] When receiving the data processing task, it is determined whether the current subtask queue is empty.

[0017] According to the current subtask queue being empty, the file path information of the to-be-processed data pointed to by the data processing task is read.

[0018] Alternatively, the data processing task is added to the task queue according to the current subtask queue being not empty, and the data processing task in the task queue waits for all subtasks in the current subtask queue to be processed before being executed.

[0019] Further, generating a data processing task based on the received task initiation instruction includes:

[0020] An immediate data processing task or a scheduled data processing task is generated based on the task initiation instruction.

[0021] The immediate data processing task is added to a task queue.

[0022] Alternatively, the scheduled data processing task is added to a task scheduled execution queue, and the scheduled data processing task is added to the task queue according to the arrival task execution time.

[0023] Furthermore, the data processing method also includes:

[0024] When generating the data processing task, field mapping information is generated based on metadata corresponding to the data to be processed.

[0025] When processing the subtask, the data to be processed is parsed according to the field mapping information read in advance.

[0026] Furthermore, the data processing method also includes:

[0027] Generate task parsing success information or task exception log information based on the parsing results of the data to be processed.

[0028] Furthermore, the data processing method also includes:

[0029] A poison pill message is generated according to the completion result of the data processing task, so as to destroy the cluster sub-role according to the poison pill message.

[0030] To achieve the above technical objectives, some embodiments of the present invention can also provide a data processing system, which may include but is not limited to a web front-end module, a scheduling service module, a cluster processing module, a data source module and a data storage module.

[0031] The web front-end module is used to generate data processing tasks based on the received task initiation instructions.

[0032] The scheduling service module is used to forward the data processing task to the cluster processing module.

[0033] Cluster processing module, including master node and slave node.

[0034] The master node is used to read the file path information of the data to be processed pointed to by the data processing task, and is used to decompose the data processing task into multiple subtasks according to the file path information. The master node is used to create a cluster subrole for processing the subtask in at least one slave node, and is used to assign at least one subtask to each of the cluster subroles.

[0035] The slave node is used to use each of the cluster sub-roles to respectively process at least one sub-task received by each of them, so as to collect and analyze the data to be processed.

[0036] Data source module, used to provide data source.

[0037] The data storage module is used to store the processed data.

[0038] In order to achieve the above-mentioned technical objectives, the present invention can also specifically provide a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the data processing method in any embodiment of the present invention.

[0039] To achieve the above-mentioned technical objectives, some embodiments of the present invention may also provide a computer storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data processing method in any embodiment of the present invention.

[0040] The beneficial effects of the present invention are:

[0041] The present invention can quickly process a large amount of data through parallel distributed data collection means, can maximize the use of computing resources, and has very high data collection and analysis efficiency. Based on the advantage of easy construction and expansion of clusters, it can better meet the needs of timely processing of rapidly growing data, greatly reducing the difficulty of data collection and analysis. The present invention can also ensure that the collected data will not be repeated, providing strong support for later data applications. The present invention has relatively low requirements on the data structure in the data source, and the present invention can more efficiently collect and parse XML structure type data, and the data processing efficiency is very high; and the present invention also has outstanding advantages such as strong scalability, high availability, and easy user operation.

[0042] The data pulling and data parsing processes in the AKKA cluster environment of the present invention both occur in the memory, avoiding the problem of conventional technology relying too much on disk performance and significantly reducing system IO (input and output processes). Therefore, the data processing efficiency of the present invention is bound to be higher and the task execution time is shorter; in particular, when the data in the data source is mostly stored in the form of small files, the advantage of the present invention in high-speed data processing is particularly prominent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic flow chart of a data processing method provided by one or more embodiments of the present invention is shown.

[0044] Figure 2 A schematic diagram showing the working principle of a data processing system provided by one or more embodiments of the present invention is shown.

[0045] Figure 3 A schematic diagram of a closed-loop control flow of task execution provided by one or more embodiments of the present invention is shown.

[0046] Figure 4 A schematic diagram showing the working principle of initiating a scheduled or immediate data processing task provided by one or more embodiments of the present invention is shown.

[0047] Figure 5 A schematic diagram of a process of transmitting and saving field mapping information provided by one or more embodiments of the present invention is shown. DETAILED DESCRIPTION

[0048] A data processing method and system, a computer device, and a computer storage medium provided by the present invention are explained and illustrated in detail below in conjunction with the drawings in the specification.

[0049] like Figure 1 As shown, and can be combined Figure 2Some embodiments of the present invention may provide a data processing method to achieve data collection and analysis, so as to solve at least one problem existing in the prior art. Specifically, the data processing method may include but is not limited to at least one of the following steps.

[0050] like Figure 2 As shown, the present invention can generate and issue task initiation instructions from the web page front-end module (WEB front-end) as an entry, and can generate data processing tasks based on the received task initiation instructions. The data to be processed involved in the present invention may include, but is not limited to, data stored in the form of XML files or other file forms. The user can enter the job name, job cycle, task type (scheduled or immediate), etc. in the WEB front-end, and select the input data source, output data terminal, metadata mapping, etc. After the output operation and selection operation are completed, the task initiation instruction can be formed by clicking to publish.

[0051] like Figure 2 , 3 4, based on the task initiation instruction, an immediate data processing task or a scheduled data processing task is generated, and the immediate data processing task is added to the task queue; or the scheduled data processing task is added to the task scheduled execution queue, and the scheduled data processing task is added to the task queue according to the arrival task execution time; more specifically, the present invention can first write the task data into the task table, Figure 4 The task table in can be implemented through MySQL, which is an open source relational database system with advantages such as stability, simple operation and maintenance, and support for transaction operations, which can work well with SpringBoot. Figure 5 As shown, when the present invention generates a data processing task, it can also generate field mapping information based on the metadata corresponding to the data to be processed, and can save the field mapping information in the form of a CSV file; specifically, the generated CSV file can be first sent from the web front-end module to the scheduling service module, and then forwarded to the cluster sub-role of the master node for storage; wherein the metadata is used to record the data structure field information in the data source, so as to clarify the specific meaning of the data field in a specific application scenario (such as a data cleaning process).

[0052] like Figure 2 As shown, the present invention can read the file path information of the data to be processed pointed to by the data processing task, for example, obtain the folder directory list from the source file FTP server. Specifically, Figure 3 , 4As shown, in some embodiments of the present invention, when a data processing task is received, it is determined whether the current subtask queue is empty: based on the current subtask queue being empty, the file path information of the to-be-processed data pointed to by the data processing task is read, or based on the current subtask queue being not empty, the data processing task is added to the task queue, and the data processing task in the task queue waits for all the subtasks in the current subtask queue to be processed before being executed. After the current data processing task is completed, the present invention can use the master node to obtain a new task from the task queue and then execute it. If the task queue is empty at this time, it waits for the arrival of the next task by receiving a new message, thus forming a closed-loop control of task execution, and continuously processing the tasks submitted by the user in sequence, such as Figure 3 shown.

[0053] Decompose the data processing task into multiple subtasks according to the file path information, specifically including but not limited to the following steps: read the upper folder path information of each file based on the file path information, use the upper folder path of a single file as the basis for task decomposition, and decompose the data processing task. During the implementation process, these upper folder path information can be added to the subtask queue, thereby realizing the decomposition of the data processing task into multiple subtasks. For example, the path corresponding to the file path information is " / data / 2020-09-01 / ", and a certain file path is " / data / 2020-09-01 / 01 / aaa.xml.zip". When the main node is used to divide the subtasks, the folder path with the same level as the upper folder path " / data / 2020-09-01 / 01 / " is used as the basis for dividing the subtasks. Of course, based on the disclosure of the present invention, at least the upper level path of the current level can also be used as the basis for task decomposition.

[0054] like Figure 2 As shown, the present invention can work in an AKKA cluster environment and create a cluster sub-role (Actor) for processing the sub-tasks; in some embodiments, the present invention can create a fixed number of cluster sub-roles, that is, one sub-role processes one or more sub-tasks. It can be understood that the cluster sub-role (Actor) can be understood as a machine for processing tasks, which has a fixed mailbox. When a message is sent to the mailbox, the Actor can read the message and process the message; wherein, when the current Actor reads the message, it can obtain the address of the sending Actor, and then can reply a result message to the sending Actor. It can be seen that the data processing based on the Actor is asynchronous and decoupled. In addition, an Actor (at this time, the parent Actor) can create its child Actor, and the parent Actor can monitor the running errors of the child Actor and set different processing methods according to the error conditions.

[0055] It is understandable that AKKA is a distributed toolset that supports fault tolerance and can support the rapid construction of large-scale distributed clusters. AKKA uses IP addresses and ports to describe each node address, so each AKKA node location is transparent to the cluster, and any AKKA node can reliably transmit messages with other nodes in the cluster based on TCP, thereby driving the reliable operation of the entire cluster system. Therefore, the present invention can avoid a large number of multi-threaded design and implementation processes in conventional solutions. Developers only need to focus on the implementation of message-driven solutions, thereby simplifying the difficulty of building parallel processing systems. Therefore, when implementing the present invention, it can also help reduce the programming workload and intensity of developers, thereby helping to reduce the development cost of the data processing system based on the present invention.

[0056] like Figure 2 As shown, the present invention can build an AKKA cluster based on the AKKA Cluster function. First, it is necessary to start the seed node 1, then start the seed node 2, and then start the computing nodes of other non-seed nodes. After the seed node is started, other nodes can join the cluster according to the current configuration path of the seed node when they are started, so the present invention can use the AKKA Cluster function provided by AKKA to quickly build a cluster. The present invention uses seed node 1 as the master node (LEADER node), and the master node is started first. The information of other related nodes started later will be monitored by the master node, so that the address information of other related nodes can be saved, so that the location of the computing node (including IP address and port) can be accurately obtained when creating a cluster sub-role. In addition, the present invention can monitor the node status information based on the JVM monitoring tool that comes with the AKKA node, such as real-time monitoring of the CPU and memory status, and can uniformly feedback the node status information monitored in real time to the master node, and the master node feeds back the node status information to the Server server for unified storage.

[0057] At least one subtask is assigned to each cluster subrole, and specifically, a subtask represented by the upper folder path information can be sent to each cluster subrole, so as to achieve the purpose of using the cluster subrole to collect and parse data. More specifically, the present invention creates communication information of all cluster subnodes based on the router function (Router) of the master node to send subtasks to each cluster subrole, and then uses each cluster subrole to process at least one subtask received by each, including pulling data from the FTP server (i.e., collecting data) based on the IP address, port and root path corresponding to the subtask, so as to collect and parse the data to be processed in the FTP server. In some embodiments of the present invention, when processing subtasks, the data to be processed is parsed according to the field mapping information read in advance, thereby eliminating the phenomenon of different interpretations after parsing the same field, and ensuring the consistency of data parsing; this method can specifically read the field mapping information in the way of reading a CSV file from the master node, and the processed data can be directly output and stored. At the same time, task parsing success information or task exception log information can be generated according to the parsing results of the data to be processed. These task parsing success information or task exception log information can be stored in the Server server and can be displayed on the WEB front end. The present invention can also generate a poison pill message according to the processing completion result of the data processing task, so as to destroy the cluster sub-role according to the poison pill message, and then recycle resources; specifically, when all sub-tasks are parsed, the master node can send a message of the completion of the current task to itself, and after receiving the message, send poison pill messages to the cluster sub-roles in all slave nodes respectively, so as to achieve the purpose of destroying the corresponding cluster sub-roles in the slave nodes. In addition, when the present invention outputs the processed data, it can determine whether there are identical files on its output path through the cluster sub-roles. If there are identical files, the file is directly deleted and then re-output. It can be seen that the present invention can avoid the problem of outputting duplicate data on the same path.

[0058] like Figure 2 As shown, the present invention can also provide a data processing system, which may include but is not limited to a web front-end module, a scheduling service module, a cluster processing module, a data source module, and a data storage module.

[0059] The web front-end module is used to generate an immediate data processing task or a scheduled data processing task based on the received task initiation instruction, which is implemented through the WEB front-end. The web front-end module can also be used to generate field mapping information based on the metadata corresponding to the data to be processed when generating data processing tasks.

[0060] The scheduling service module is used to immediately forward the immediate data processing task to the cluster processing module, or add the scheduled data processing task to the task timing execution queue, and forward the scheduled data processing task to the cluster processing module according to the arrival task execution time. The SpringBoot specifically involved in the present invention is a J2EE framework that can quickly complete the deployment and development of the server program, and then can realize the integrated AKKA function, generate a task sending node and its sub-roles, and the present invention can be based on SpringBoot to specifically deploy and develop server programs, shorten the project cycle and save project expenses. The scheduling service module of the present invention can be implemented through the Server server program. Based on the Server server program, the present invention can not rely on Linux timing scripts and third-party scheduling tools when implementing scheduled tasks. It only needs to specify the task type as a scheduled task and select the task execution time when creating a task, so the present invention effectively simplifies the system deployment dependency and reduces the system learning and use costs. Some embodiments of the present invention can deploy both the web front-end module and the scheduling service module on a Linux server.

[0061] The cluster processing module includes a master node (LEADER node) and a slave node (computing node). The master node can be used to add an immediate data processing task to a task queue or to add a scheduled data processing task to a task queue when the task execution time is reached. The slave node used to provide system computing resources can be used to deploy corresponding cluster sub-roles, and a slave node involved in the present invention can be implemented by a physical server.

[0062] The master node can be used to read the file path information of the data to be processed pointed to by the data processing task, and to decompose the data processing task into multiple subtasks according to the file path information. In some embodiments of the present invention, the master node is specifically used to read the upper folder path information of each file based on the file path information, and is used to use the upper folder path of a single file as the basis for task decomposition, so as to decompose the data processing task and realize the decomposition of the data processing task into multiple subtasks. The master node is used to create a certain number of cluster sub-roles for processing subtasks in at least one slave node. The specific number of cluster sub-roles of the present invention can be set in the configuration file, and is used to assign at least one subtask to each cluster sub-role respectively. It can be understood that the number of cluster sub-roles generated by the present invention is much smaller than the number of subtasks by default. After parsing a subtask, a cluster sub-role will send a message that the current subtask parsing is completed to the master node, and then the master node can take out a new subtask from the subtask queue and send it to the cluster sub-role. In addition, some embodiments of the present invention can arrange the master node through a separately set server, and the server may include but is not limited to Linux servers.

[0063] The slave node is used to utilize each cluster sub-role to process at least one sub-task received by each of them respectively, so as to collect and analyze the data to be processed. The master node can also be used to determine whether the current sub-task queue is empty when receiving the data processing task, and to read the file path information of the data to be processed pointed to by the data processing task according to the current sub-task queue being empty, or to add the data processing task to the task queue according to the current sub-task queue being not empty, and the data processing task in the task queue waits for the sub-tasks in the current sub-task queue to be processed and then executed. When utilizing each cluster sub-role to process the sub-task, the data to be processed can be analyzed according to the field mapping information read in advance. The cluster sub-role can also be used to generate task analysis success information or task abnormality log information according to the analysis result of the data to be processed, and send the generated analysis success information or task abnormality log information to the master node for unified processing. Some embodiments of the present invention can also arrange at least one slave node through a Linux server, for example, arrange a slave node on a Linux server.

[0064] In addition, the master node is also used to generate a poison pill message based on the completion result of the data processing task, and to send the poison pill information to the cluster sub-role in the slave node. The cluster sub-role used to execute the current task is destroyed according to the received poison pill message to achieve the purpose of recycling the corresponding computing resources.

[0065] The data source module is used to provide a data source, which can be implemented through an FTP server cluster. The present invention can set the data source name, IP address, port, access account, access password, root path and other information based on the web front-end module to collect and analyze the data in the data source of the corresponding path.

[0066] The data storage module is used to store the processed data, for example, through an FTP server, a HDFS server, or a HIVE server.

[0067] It is understandable that the present invention can also provide a computer device, which includes a memory and a processor, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the data processing method in any embodiment of the present invention. The data processing method may include but is not limited to at least one of the following steps: generating a data processing task based on the received task initiation instruction. Specifically, an immediate data processing task or a scheduled data processing task is generated based on the task initiation instruction, and the immediate data processing task is added to the task queue; or the scheduled data processing task is added to the task timing execution queue, and the scheduled data processing task is added to the task queue according to the arrival task execution time. In addition, when the present invention generates a data processing task, field mapping information can be generated based on the metadata corresponding to the data to be processed. Read the file path information where the data to be processed pointed to by the data processing task is located. In some embodiments of the present invention, when receiving a data processing task, it is determined whether the current subtask queue is empty: according to the current subtask queue being empty, the file path information where the data to be processed pointed to by the data processing task is located, or according to the current subtask queue being not empty, the data processing task is added to the task queue, and the data processing task in the task queue waits for the subtasks in the current subtask queue to be processed before being executed. Decomposing a data processing task into multiple subtasks according to the file path information specifically includes the following steps: reading the upper folder path information of each file based on the file path information; using the upper folder path of a single file as the basis for task decomposition, decomposing the data processing task, and decomposing the data processing task into multiple subtasks. Creating a cluster subrole for processing the subtasks, and assigning at least one subtask to each cluster subrole. Using each cluster subrole to process at least one subtask received by each cluster subrole, respectively, to collect and parse the data to be processed. The present invention can generate task parsing success information or task exception log information according to the parsing results of the data to be processed. When processing subtasks, some embodiments of the present invention parse the data to be processed according to the pre-read field mapping information, and can store the processed data. The present invention can also generate a poison pill message according to the completion result of the data processing task processing, so as to destroy the cluster subrole according to the poison pill message, and then recycle resources.

[0068] It is understandable that the present invention may provide a computer storage medium, on which computer readable instructions are stored, and when the computer readable instructions are executed by one or more processors, one or more processors execute the steps of the data processing method in any embodiment of the present invention. The data processing method may include but is not limited to at least one of the following steps: generating a data processing task based on the received task initiation instruction. Specifically, an immediate data processing task or a scheduled data processing task is generated based on the task initiation instruction, and the immediate data processing task is added to the task queue; or the scheduled data processing task is added to the task timing execution queue, and the scheduled data processing task is added to the task queue according to the arrival task execution time. In addition, when the present invention generates a data processing task, field mapping information can be generated based on the metadata corresponding to the data to be processed. Read the file path information where the data to be processed pointed to by the data processing task is located. In some embodiments of the present invention, when receiving a data processing task, it is determined whether the current subtask queue is empty: according to the current subtask queue being empty, the file path information where the data to be processed pointed to by the data processing task is located, or according to the current subtask queue being not empty, the data processing task is added to the task queue, and the data processing task in the task queue waits for all the subtasks in the current subtask queue to be processed before being executed. Decomposing a data processing task into multiple subtasks according to the file path information specifically includes the following steps: reading the upper folder path information of each file based on the file path information; using the upper folder path of a single file as the basis for task decomposition, decomposing the data processing task, and decomposing the data processing task into multiple subtasks. Creating a cluster subrole for processing the subtasks, and assigning at least one subtask to each cluster subrole. Using each cluster subrole to process at least one subtask received by each cluster subrole, respectively, to collect and parse the data to be processed. The present invention can generate task parsing success information or task exception log information according to the parsing results of the data to be processed. When processing subtasks, some embodiments of the present invention parse the data to be processed according to the pre-read field mapping information, and can store the processed data. The present invention can also generate a poison pill message according to the completion result of the data processing task processing, so as to destroy the cluster subrole according to the poison pill message, and then recycle resources.

[0069] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute instructions), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable storage medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device. More specific examples of computer-readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic devices), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory, or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM, Compact Disc Read-Only Memory). In addition, the computer-readable storage medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0070] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA, Programmable Gate Array), a field programmable gate array (FPGA, Field Programmable Gate Array), etc.

[0071] In the description of this specification, the description with reference to the terms "this embodiment", "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0072] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and simple improvements made to the essential contents of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: generating a data processing task based on the received task initiation instruction; Read the file path information of the data to be processed pointed to by the data processing task; Decomposing the data processing task into a plurality of subtasks according to the file path information; Creating a cluster subrole for processing the subtask; Allocating at least one subtask to each of the cluster subroles; Using each of the cluster sub-roles to respectively process at least one sub-task received by each of them, so as to collect and analyze the data to be processed; Decomposing the data processing task into a plurality of subtasks according to the file path information comprises: Reading the upper folder path information of each file based on the file path information; The data processing task is decomposed based on the upper folder path of a single file as the basis for task decomposition, and the data processing task is decomposed into multiple subtasks.

2. The data processing method according to claim 1, characterized in that: Also includes: Upon receiving the data processing task, determining whether the current subtask queue is empty; According to the current subtask queue being empty, read the file path information of the to-be-processed data pointed to by the data processing task; Alternatively, the data processing task is added to the task queue according to the current subtask queue being not empty, and the data processing task in the task queue waits for all subtasks in the current subtask queue to be processed before being executed.

3. The data processing method according to claim 2, characterized in that: Generating a data processing task based on the received task initiation instruction includes: Generate an immediate data processing task or a scheduled data processing task based on the task initiation instruction; Adding the immediate data processing task to a task queue; Alternatively, the scheduled data processing task is added to a task scheduled execution queue, and the scheduled data processing task is added to the task queue according to the arrival task execution time.

4. The data processing method according to claim 1, characterized in that: Also includes: When generating the data processing task, generating field mapping information based on metadata corresponding to the data to be processed; When processing the subtask, the data to be processed is parsed according to the field mapping information read in advance.

5. The data processing method according to claim 4, characterized in that: Also includes: Generate task parsing success information or task exception log information based on the parsing results of the data to be processed.

6. The data processing method according to claim 1, characterized in that: Also includes: A poison pill message is generated according to the completion result of the data processing task, so as to destroy the cluster sub-role according to the poison pill message.

7. A data processing system, characterized in that: include: A web front-end module, used to generate data processing tasks based on received task initiation instructions; A scheduling service module, used for forwarding the data processing task to the cluster processing module; Cluster processing module, including master node and slave node; The master node is used to read the file path information of the data to be processed pointed to by the data processing task, and to decompose the data processing task into multiple subtasks according to the file path information; The master node is used to create a cluster sub-role for processing the sub-task in at least one slave node, and to allocate at least one sub-task to each of the cluster sub-roles; The slave node is used to use each of the cluster sub-roles to respectively process at least one sub-task received by each of them, so as to collect and analyze the data to be processed; Data source module, used to provide data source; A data storage module, used for storing processed data; Decomposing the data processing task into a plurality of subtasks according to the file path information comprises: Reading the upper folder path information of each file based on the file path information; The data processing task is decomposed based on the upper folder path of a single file as the basis for task decomposition, and the data processing task is decomposed into multiple subtasks.

8. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the data processing method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that: The computer storage medium stores computer-readable instructions, which, when executed by one or more processors, enable the one or more processors to perform the steps of the data processing method according to any one of claims 1 to 6.

Images