A data processing method and related equipment

Abstract

The present application provides a data processing method and related equipment, which can realize high-concurrency processing of computing tasks corresponding to data frames and improve computing efficiency. The method includes: step 1, determining the current N idle computing nodes in the distributed cluster; step 2, obtaining N task messages from the target message queue through the N idle computing nodes; step 3, determining the N data frames corresponding to the N task messages and the computing logic corresponding to the N task messages through the N idle computing nodes; step 4, based on the N idle computing nodes, the data in the N data frames are calculated through the computing logic corresponding to the N task messages; when there is a first idle computing node that has completed the calculation among the N idle computing nodes, and there is an uncalculated first task message in the target message queue, repeating steps 2 to 4 based on the first idle computing node until the task message calculation in the target message queue is completed.

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a data processing method and related equipment. Background Art

[0002] Hadoop is an open source big data framework and a distributed computing solution. The two core components of Hadoop solve the data storage problem, namely the distributed file system (Hadoop Distributed File System, HDFS) and the distributed computing problem (MapReduce). The general working process is as follows: when submitting a computing task to the MapReduce framework, it will first split the computing task into several Map tasks, and then distribute them to different nodes for execution. Each Map task processes a part of the input data. When the Map task is completed, it will generate some intermediate files, which will be used as the input data of the Reduce task. The main goal of the Reduce task is to aggregate the outputs of the previous Maps and output them.

[0003] The above method is used for big data computing scenarios, where the intermediate results of Hadoop's mapreduce are stored in HDFS. During the computing process, tasks need to be initiated to the master node, and then the master node splits and distributes the tasks to the slave nodes for computing. During this period, the tasks cannot be executed concurrently, and the next task can only be executed after one task is completed. Summary of the invention

[0004] The present application provides a data processing method and related equipment, which can realize high-concurrency computing of computing tasks and improve computing efficiency.

[0005] The first aspect of the present application provides a data processing method, comprising:

[0006] Step 1: Determine the current N idle computing nodes in the distributed cluster, where N is a positive integer greater than or equal to 1;

[0007] Step 2: Obtain N task messages from a target message queue through the N idle computing nodes, the N idle computing nodes correspond to the N task messages, the target message queue contains M task messages to be processed by the user, and the N task messages are included in the M task messages;

[0008] Step 3: Determine N data frames corresponding to the N task messages and computing logic corresponding to the N task messages through the N idle computing nodes;

[0009] Step 4: Based on the N idle computing nodes, the data in the N data frames are calculated by the computing logic corresponding to the N task messages;

[0010] When there is a first idle computing node that has completed calculation among the N idle computing nodes, and there is an uncalculated first task message in the target message queue, steps 2 to 4 are repeated based on the first idle computing node until the calculation of the task message in the target message queue is completed.

[0011] Optionally, the calculating the data of the N data frames based on the N idle computing nodes by using a computing logic corresponding to the N task messages includes:

[0012] Optimizing and splitting the computing logic corresponding to the N task messages through the N idle computing nodes to obtain the split computing logic;

[0013] Based on the second idle computing node, the data in the first data frame is calculated by the first computing logic, the first computing logic is any one of the split computing logics, the first data frame is the data frame corresponding to the first computing logic among the N data frames, and the second idle computing node is the idle computing node corresponding to the first data frame among the N idle computing nodes.

[0014] Optionally, determining, by means of the N idle computing nodes, N data frames corresponding to the N task messages and computing logic corresponding to the N task messages includes:

[0015] The target task message is parsed by the target idle computing node to obtain the storage address of the target data frame corresponding to the target task message and the computing logic of the target data frame, wherein the target task message is any one of the N task messages, and the target idle computing node is a computing node corresponding to the target task message among the N idle computing nodes;

[0016] The target data frame is acquired according to the storage address of the target data frame.

[0017] Optionally, the method further comprises:

[0018] Receive target operation instructions from the user;

[0019] A target message queue is generated according to the target operation instruction.

[0020] Optionally, the method further comprises:

[0021] When there is a first idle computing node that has completed calculation among the N idle computing nodes, the calculation result corresponding to the first idle computing node is stored, and the task calculation in the task message corresponding to the first idle computing node is marked as completed.

[0022] A second aspect of the present application provides a data processing device, including:

[0023] A first determining unit is used to execute step 1, determine N current idle computing nodes in the distributed cluster, where N is a positive integer greater than or equal to 1;

[0024] An acquisition unit, configured to execute step 2, acquire N task messages from a target message queue through the N idle computing nodes, the N idle computing nodes corresponding to the N task messages, the target message queue containing the M task messages to be processed by the user, and the N task messages being contained in the M task messages;

[0025] A second determining unit, configured to execute step 3, determining N data frames corresponding to the N task messages and computing logics corresponding to the N task messages through the N idle computing nodes;

[0026] A computing unit, used in step 4, based on the N idle computing nodes, to compute the data in the N data frames by using computing logic corresponding to the N task messages;

[0027] A processing unit is used to repeat steps 2 to 4 based on a first idle computing node that has completed calculations, when there is a first idle computing node among the N idle computing nodes and there is a first task message that has not been calculated in the target message queue, until the calculation of the task message in the target message queue is completed.

[0028] Optionally, the computing unit is specifically used for:

[0029] Optimizing and splitting the computing logic corresponding to the N task messages through the N idle computing nodes to obtain the split computing logic;

[0030] Based on the second idle computing node, the data in the first data frame is calculated by the first computing logic, the first computing logic is any one of the split computing logics, the first data frame is the data frame corresponding to the first computing logic among the N data frames, and the second idle computing node is the idle computing node corresponding to the first data frame among the N idle computing nodes.

[0031] Optionally, the second determining unit is specifically configured to:

[0032] The target task message is parsed by the target idle computing node to obtain the storage address of the target data frame corresponding to the target task message and the computing logic of the target data frame, wherein the target task message is any one of the N task messages, and the target idle computing node is a computing node corresponding to the target task message among the N idle computing nodes;

[0033] The target data frame is acquired according to the storage address of the target data frame.

[0034] Optionally, the processing unit is further configured to:

[0035] Receive target operation instructions from the user;

[0036] A target message queue is generated according to the target operation instruction.

[0037] Optionally, the data processing device further includes:

[0038] The storage unit is used to store the calculation result corresponding to the first idle computing node when there is a first idle computing node that has completed the calculation among the N idle computing nodes, and mark the task calculation in the task message corresponding to the first idle computing node as completed.

[0039] A third aspect of the present application provides a computer device, which includes at least one connected processor and memory, wherein the memory is used to store program code, and the program code is loaded and executed by the processor to implement the steps of the data processing method described above.

[0040] A fourth aspect of the present application provides a computer-readable storage medium, which includes instructions, which, when executed on a computer, enable the computer to execute the steps of the data processing method described above.

[0041] In summary, it can be seen that in the embodiments provided by the present application, the computing data to be dealt with is a data frame, and one unit of data can be completed in one computing node. In a distributed cluster, its computing task can achieve high-concurrency computing, and when the task calculation in the computing node is completed, the task message can be repeatedly obtained and the calculation can be performed. It is not like the existing master node that splits and distributes the task to the slave node for calculation. During the period, its tasks cannot be executed concurrently, and it is necessary to wait for one task to be completed before executing the next task, thereby improving the efficiency of the calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a computing architecture of a data processing method provided in an embodiment of the present application;

[0043] Figure 2A flowchart of a data processing method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a virtual structure of a data processing device provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the hardware structure of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0047] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than the content illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or modules need not be limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules that appear in this application is only a logical division. There may be other division modes when implemented in practical applications, for example, multiple modules may be combined into or integrated in another system, or some feature vectors may be ignored, or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.

[0048] Some of the terms involved in this application are explained:

[0049] Data frame: refers to a group of data consisting of "frame header + data information", such as the table structure description and the composition of table data in the database.

[0050] Distributed: refers to the distribution of computing resources in different physical nodes.

[0051] High concurrency: refers to the design that ensures that the system can process many requests in parallel. In layman's terms, high concurrency means that many users access a resource at the same time.

[0052] NumPy: is an open source numerical computing extension for Python. This tool can be used to store and process large matrices. It is much more efficient than Python's own nested list structure, supports a large number of dimensional arrays and matrix operations, and also provides a large number of mathematical function libraries for array operations.

[0053] Pandas: is a powerful tool set for analyzing structured data; it is based on NumPy (which provides high-performance matrix operations); it is used for data mining and data analysis, and also provides data cleaning functions.

[0054] This application is based on the distributed task architecture concept and the Pandas computing library. The data object processed by its computing tasks is a data frame with small batches of data. The effect achieved is a lightweight framework for efficient concurrent computing of data frames. Figure 1 , Figure 1 The computing framework diagram of the data processing method provided in the embodiment of the present application includes a task 101, a message queue 102, a computing node 103 and a temporary storage center for results 104, wherein task 101 includes tasks 1 to N, each computing task includes a data frame to be processed and its corresponding computing logic (wherein the computing logic is a set of operations on the data in the computing task, for example, adding class A data and class B data to obtain a value, or finding the variance of certain columns in the data to generate new data, and of course there are other operations, which are not specifically limited); the message queue 102 is a queue for storing task messages; the computing node 103 is a queue for storing task messages; the computing node 104 ... 103 is a distributed Pandas computing node, including computing node 1 to computing node N, wherein a computing node obtains a task message and performs corresponding calculations, specifically: receiving the task message, parsing the data frame storage address in the task message, and obtaining the data frame from the data frame address, parsing the calculation logic in the task message, and extracting data from the data frame to perform calculations to obtain calculation results; the result temporary storage center 104 is used to temporarily store the calculation results obtained by the computing node on the data in the data frame. In the actual use process, the result after the task calculation can be obtained from the result temporary storage center through the task ID.

[0055] The data processing method provided in the present application is described below from the perspective of a data processing device. The data processing device can be a terminal device, a server, or a service unit in a server, without specific limitation.

[0056] See also Figure 2 , Figure 2 A flow chart of a data processing method provided in an embodiment of the present application includes:

[0057] 201. Determine the current N idle computing nodes in the distributed cluster.

[0058] In this embodiment, the data processing device can determine the current N idle computing nodes in the distributed cluster, where N is a positive integer greater than or equal to 1, and the number of computing nodes included in the distributed cluster is greater than or equal to 2. That is, the data processing device here can determine the current idle computing nodes in the distributed cluster.

[0059] 202. Obtain N task messages from the target message queue through N idle computing nodes.

[0060] In this embodiment, after determining the current idle computing node in the distributed cluster, the data processing device can obtain N task messages from the target message queue through N idle computing nodes, wherein the target message queue contains M task messages to be processed by the user, and the N task messages are included in the M task messages, and the N idle computing nodes correspond to the N task messages, that is, the target message queue stores multiple task messages, each task message has a corresponding identifier, and another idle computing node distributes a task message (specifically, the N task messages can be distributed to the N idle computing nodes according to preset rules, for example, the preset rules can be first come first served, which idle computing node sends the request first to send the task message to the computing node first, or N task messages are sent to N idle computing nodes at one time, each computing node can get a task message and perform corresponding calculations), and a task message is distributed to each of the N idle computing nodes.

[0061] In one embodiment, the data processing device may receive a target operation instruction from a user, and generate a target message queue according to the target operation instruction.

[0062] In this embodiment, the data processing device can generate a target message queue based on the user's target operation instructions. That is, the user determines the data to be used, the storage address of the data, and what kind of calculation needs to be done with the data through a visual operation interface, and generates a task message and stores it in the target message queue.

[0063] In one embodiment, the operation for generating the target operation instruction includes at least one of a gesture operation, a sliding operation, a click operation, and a voice control operation. For example, when a user performs a click operation on a visual operation interface, a data processing device may receive the click operation. At this time, the click operation generates the target operation instruction. That is to say, the operation instruction may be defined in advance. For example, a sliding operation may be defined in advance as an operation for generating a target message queue (such as a left slide operation, a right slide operation, an up slide operation, and a down slide operation, etc.), or a click operation may be defined as an operation for generating a target message queue (such as a single-click operation or a double-click operation, etc.), or a gesture operation may be defined as an operation for generating a target message queue (such as swinging the wrist or arm to the left, swinging the wrist or arm to the right, such as a four-finger pinch operation or a three-finger up slide operation, etc.), or a voice control operation may be defined as an operation for generating a target message queue (such as receiving the sound of generating a target message queue and the sound of a specific task message in the target message queue). The above is only an example and does not represent a limitation on the operation for generating a target operation instruction.

[0064] 203. Determine N data frames corresponding to the N task messages and computing logics corresponding to the N task messages through N idle computing nodes.

[0065] In this embodiment, after sending N task messages to N idle computing nodes, the data processing device can determine N data frames corresponding to the N task messages and computing logics corresponding to the N task messages through the N idle computing nodes.

[0066] In one embodiment, the data processing device determines N data frames corresponding to N task messages and the computing logic corresponding to the N task messages through N idle computing nodes, including:

[0067] The target task message is parsed through the target idle computing node to obtain the storage address of the target data frame corresponding to the target task message and the computing logic of the target data frame, the target task message is any one of the N task messages, and the target idle computing node is the computing node corresponding to the target task message among the N idle computing nodes;

[0068] The target data frame is obtained according to the storage address of the target data frame.

[0069] In this embodiment, the data processing device can parse N task messages through N idle computing nodes respectively to obtain the storage address of the data frame carried in the task message and the calculation logic of the data frame. Specifically, the target task message can be parsed through the target idle computing node to obtain the storage address of the target data frame corresponding to the target task message and the calculation logic of the target data frame, wherein the target task message is any one of the N task messages, and the target idle computing node is the computing node corresponding to the target task message among the N idle computing nodes. That is to say, here, the data processing device can parse the task message sent to the idle computing node through the idle computing node to obtain the storage address of the data frame carried in the task message (that is, where the data frame is stored) and the calculation logic corresponding to the data frame, and then the data frame can be obtained according to the storage address of the data frame. It can be understood that the data frame in this application can be data of an order of magnitude of less than 500,000 rows, and of course it can also be data of other orders of magnitude, such as data of less than 600,000 rows or 700,000 rows, which is not specifically limited. In addition, the task message may be sent to an idle computing node, or may be sent after the idle computing node sends a request, and there is no specific limitation.

[0070] It should be noted that the task message may also include a method for reading the data frame, that is, the storage medium where the data frame is stored can be known through the reading method, and then the corresponding driver is called to read the data frame.

[0071] 204. Based on the N idle computing nodes, the data in the N data frames are calculated by the computing logic corresponding to the N task messages.

[0072] In this embodiment, the data processing device can calculate the data in N data frames based on N idle computing nodes through the computing logic corresponding to the N task messages. That is, after obtaining the data frame corresponding to the task and the corresponding computing logic, the data in the data frame can be calculated through the computing logic.

[0073] In one embodiment, the data processing device calculates the data of N data frames based on N idle computing nodes by using computing logic corresponding to N task messages, including:

[0074] The computing logic corresponding to the N task messages is optimized and split by using N idle computing nodes to obtain the split computing logic;

[0075] Based on the second idle computing node, the data in the first data frame is calculated by the first computing logic, the first computing logic is any one of the split computing logics, the first data frame is the data frame corresponding to the first computing logic among the N data frames, and the second idle computing node is the idle computing node corresponding to the first data frame among the N idle computing nodes.

[0076] In this embodiment, the data processing device may first optimize and split the computing logic in the task message through the idle computing nodes to obtain the split computing logic, that is, optimize the computing logic in the task message to facilitate the idle computing nodes to perform fast calculations; then, based on the second idle computing node, the data in the first data frame is calculated through the first computing logic. In other words, the data processing device can match the computing function of the data analysis library according to the split computing logic. The embodiment of the present invention adopts Pandas and then uses Pandas to perform fast calculations, wherein the calculation data and calculation process of Pandas occur in the physical memory of the computer. Since the computing framework of Pandas is deployed in the memory, Pandas will pull the task data into the memory for calculation, which can improve the efficiency of parallel processing.

[0077] 205. When there is a first idle computing node that has completed calculation among the N idle computing nodes, and there is an uncalculated first task message in the target message queue, steps 202 to 204 are repeatedly executed based on the first idle computing node until the calculation of the task message in the target message queue is completed.

[0078] In this embodiment, since the calculation time of the data frames in different task messages may be different, this may result in that after some idle computing nodes among the N idle computing nodes have completed the calculation of the data frames in the task messages corresponding to them, other idle computing nodes among the N idle computing nodes have not yet completed the calculation, and there are still uncalculated task messages in the target message queue. At this time, the idle computing nodes that have completed the calculation can repeat steps 202 to 204 until the calculation of the task messages in the target message queue is completed, that is, when an idle computing node has completed the calculation, the task message is obtained from the target message queue, and then the obtained task message is parsed by the idle computing node to obtain the data frame storage address and its corresponding calculation logic, and the data frame is obtained by the idle computing node, and the data frame is calculated by the calculation logic, and so on, until all the task messages in the target message queue are calculated.

[0079] In one embodiment, when there is a first idle computing node that has completed calculations among the N idle computing nodes, the calculation result corresponding to the first idle computing node is stored, and the task calculation completion in the task message corresponding to the first idle computing node is marked.

[0080] In this embodiment, the data processing can associate the calculation result corresponding to the completed task node with its corresponding task ID and store it in the result temporary storage center, and mark the task calculation completion in the task message corresponding to the idle computing node, so that even during the calculation process of the task, the user can view the calculation result of the task through the completed task ID. It is understandable that after obtaining the calculation result, the calculation result can also be directly displayed to the user for viewing.

[0081] In summary, it can be seen that in the embodiments provided by the present application, the computing data to be dealt with is a data frame, and one unit of data can be completed in one computing node. In a distributed cluster, its computing task can achieve high-concurrency computing, and when the task calculation in the computing node is completed, the task message can be repeatedly obtained and the calculation can be performed. It is not like the existing master node that splits and distributes the task to the slave node for calculation. During the period, its tasks cannot be executed concurrently, and it is necessary to wait for one task to be completed before executing the next task, thereby improving the efficiency of the calculation.

[0082] The present application is described above from the perspective of a data processing method, and the present application is described below from the perspective of a data processing device.

[0083] See also Figure 3 , Figure 3 A virtual structure diagram of a data processing device provided in an embodiment of the present application includes:

[0084] The first determining unit 301 is used to execute step 1, determine the current N idle computing nodes in the distributed cluster, where N is a positive integer greater than or equal to 1;

[0085] The acquisition unit 302 is used to execute step 2, obtain N task messages from the target message queue through the N idle computing nodes, the N idle computing nodes correspond to the N task messages, the target message queue contains the M task messages to be processed by the user, and the N task messages are included in the M task messages;

[0086] The second determining unit 303 is used to execute step 3, determine N data frames corresponding to the N task messages and computing logics corresponding to the N task messages through the N idle computing nodes;

[0087] The calculation unit 304 is used for step 4, based on the N idle computing nodes, calculating the data in the N data frames by the calculation logic corresponding to the N task messages;

[0088] The processing unit 305 is used to repeat steps 2 to 4 based on the first idle computing node when there is a first idle computing node that has completed calculation among the N idle computing nodes and there is a first task message that has not been calculated in the target message queue, until the calculation of the task message in the target message queue is completed.

[0089] Optionally, the calculation unit 304 is specifically configured to:

[0090] Optimizing and splitting the computing logic corresponding to the N task messages through the N idle computing nodes to obtain the split computing logic;

[0091] Based on the second idle computing node, the data in the first data frame is calculated by the first computing logic, the first computing logic is any one of the split computing logics, the first data frame is the data frame corresponding to the first computing logic among the N data frames, and the second idle computing node is the idle computing node corresponding to the first data frame among the N idle computing nodes.

[0092] Optionally, the second determining unit 303 is specifically configured to:

[0093] The target task message is parsed by the target idle computing node to obtain the storage address of the target data frame corresponding to the target task message and the computing logic of the target data frame, wherein the target task message is any one of the N task messages, and the target idle computing node is a computing node corresponding to the target task message among the N idle computing nodes;

[0094] The target data frame is acquired according to the storage address of the target data frame.

[0095] Optionally, the processing unit 305 is further configured to:

[0096] Receive target operation instructions from the user;

[0097] A target message queue is generated according to the target operation instruction.

[0098] Optionally, the data processing device further includes:

[0099] The storage unit 306 is used to store the calculation result corresponding to the first idle computing node when there is a first idle computing node that has completed the calculation among the N idle computing nodes, and mark the task calculation in the task message corresponding to the first idle computing node as completed.

[0100] Figure 4 4 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 422 (for example, one or more processors) and a memory 432, and one or more storage media 430 (for example, one or more mass storage devices) storing application programs 442 or data 444. Among them, the memory 432 and the storage medium 430 may be short-term storage or permanent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 422 may be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the server 400.

[0101] The server 400 may also include one or more power supplies 426, one or more wired or wireless network interfaces 450, one or more input and output interfaces 458, and / or one or more operating systems 441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0102] The steps performed by the data processing device in the above embodiment can be based on the Figure 4 The server structure shown.

[0103] An embodiment of the present application also provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0104] An embodiment of the present application further provides a processor, which is used to run a program, wherein the program executes the steps of the above-mentioned data processing method when running.

[0105] An embodiment of the present application also provides a terminal device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program code is loaded and executed by the processor to implement the steps of the above-mentioned data processing method.

[0106] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the above-mentioned data processing method.

[0107] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0118] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Step 1: Determine the current N idle computing nodes in the distributed cluster, where N is a positive integer greater than or equal to 1; Step 2: Obtain N task messages from the target message queue through the N idle computing nodes, the N idle computing nodes correspond to the N task messages, the target message queue contains M task messages to be processed by the user, and the N task messages are included in the M task messages; The task message carries the storage address of the data frame and the calculation logic of the data frame; Step 3: Determine N data frames corresponding to the N task messages and computing logic corresponding to the N task messages through the N idle computing nodes; Step 4: Based on the N idle computing nodes, the data in the N data frames are calculated by the computing logic corresponding to the N task messages; wherein, based on the N idle computing nodes, the data in the N data frames are calculated by the computing logic corresponding to the N task messages, including: optimizing and splitting the computing logic corresponding to the N task messages by the N idle computing nodes to obtain the split computing logic; based on the second idle computing node, the data in the first data frame is calculated by the first computing logic, the first computing logic is any one of the split computing logics, the first data frame is the data frame corresponding to the first computing logic in the N data frames, and the second idle computing node is the idle computing node corresponding to the first data frame in the N idle computing nodes; When there is a first idle computing node that has completed calculation among the N idle computing nodes, and there is an uncalculated first task message in the target message queue, steps 2 to 4 are repeated based on the first idle computing node until the calculation of the task message in the target message queue is completed.

2. The method according to claim 1, characterized in that The determining, by the N idle computing nodes, N data frames corresponding to the N task messages and computing logic corresponding to the N task messages comprises: The target task message is parsed by the target idle computing node to obtain the storage address of the target data frame corresponding to the target task message and the computing logic of the target data frame, wherein the target task message is any one of the N task messages, and the target idle computing node is a computing node corresponding to the target task message among the N idle computing nodes; The target data frame is acquired according to the storage address of the target data frame.

3. The method according to any one of claims 1 to 2, characterized in that The method further comprises: Receive target operation instructions from the user; A target message queue is generated according to the target operation instruction.

4. The method according to any one of claims 1 to 2, characterized in that The method further comprises: When there is a first idle computing node that has completed calculation among the N idle computing nodes, the calculation result corresponding to the first idle computing node is stored, and the task calculation in the task message corresponding to the first idle computing node is marked as completed.

5. A data processing device, characterized in that: include: A first determining unit is used to execute step 1, determine N current idle computing nodes in the distributed cluster, where N is a positive integer greater than or equal to 1; An acquisition unit, configured to execute step 2, acquire N task messages from a target message queue through the N idle computing nodes, the N idle computing nodes corresponding to the N task messages, the target message queue containing M task messages to be processed by the user, and the N task messages being contained in the M task messages; The task message carries the storage address of the data frame and the calculation logic of the data frame; A second determining unit, configured to execute step 3, determining N data frames corresponding to the N task messages and computing logics corresponding to the N task messages through the N idle computing nodes; A computing unit, used for step 4, based on the N idle computing nodes, calculating the data in the N data frames by the computing logic corresponding to the N task messages; wherein the computing unit is specifically used for: optimizing and splitting the computing logic corresponding to the N task messages by the N idle computing nodes to obtain the split computing logic; based on the second idle computing node, calculating the data in the first data frame by the first computing logic, the first computing logic being any one of the split computing logics, the first data frame being a data frame corresponding to the first computing logic in the N data frames, and the second idle computing node being an idle computing node corresponding to the first data frame in the N idle computing nodes; A processing unit is used to repeat steps 2 to 4 based on a first idle computing node that has completed calculations, when there is a first idle computing node among the N idle computing nodes and there is a first task message that has not been calculated in the target message queue, until the calculation of the task message in the target message queue is completed.

6. The device according to claim 5, characterized in that The second determining unit is specifically configured to: The target task message is parsed by the target idle computing node to obtain the storage address of the target data frame corresponding to the target task message and the computing logic of the target data frame, wherein the target task message is any one of the N task messages, and the target idle computing node is a computing node corresponding to the target task message among the N idle computing nodes; The target data frame is acquired according to the storage address of the target data frame.

7. A computer device, characterized in that: include: At least one connected processor and memory, wherein the memory is used to store program code, and the program code is loaded and executed by the processor to implement the steps of the data processing method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the steps of the data processing method according to any one of claims 1 to 4.

Images