Data processing methods, unified data processors, and readable storage media

Abstract

This application relates to the field of computer technology and provides a data processing method, a unified data processor, and a readable storage medium. The method, applied to a unified data processor, includes: receiving a computing task sent by an application system through a main management node and determining computing information based on the computing task; determining a first target computing node from multiple computing nodes through the main management node based on the computing task information and resource information of each computing node, and sending the computing information to the first target computing node; and obtaining initial data from multiple computing nodes and / or multiple data sources through the first target computing node according to the computing task, and performing calculations based on the initial data to complete the computing task. This method can reduce the data processing pressure on the application system and minimize the impact of data processing on business database data storage operations.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a data processing method, a unified data processor, and a readable storage medium. Background Technology

[0002] Amid rising labor costs and an aging population, coupled with the development of robotics technology, the market demand for commercial service robots continues to surge, and the number of commercial service robots put into use is increasing daily. With the continuous use of numerous commercial service robots, a large amount of robot data is generated daily, which is then reported to business databases or file servers. Different types of robot data are typically distributed across multiple different business databases.

[0003] Currently, when analyzing robot data, application systems or clients directly query the business database to obtain the data, and then perform analysis and calculations on the obtained data. This operation requires the application system or client to first pull data from the business database via the network and then perform local calculations, resulting in low data processing efficiency. Furthermore, due to the limited local computing resources of the application system or client, it may crash when faced with massive amounts of robot data. In addition, the large amount of data reads from the business database by the application system or client can also affect the data storage of the business database.

[0004] Therefore, how to analyze and process massive amounts of robot data from multiple business databases while minimizing the impact on data storage in business databases and ensuring the smooth operation of application systems or clients has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a data processing method, a unified data processor, and a readable storage medium.

[0006] In a first aspect, embodiments of this application provide a data processing method applied to a unified data processor. The unified data processor includes multiple computing nodes and at least one management node. The at least one management node includes a master management node, which is connected to an application system through a unified interface. The master management node stores resource information of each computing node, and the multiple computing nodes synchronously store data from multiple data sources. The method includes:

[0007] The main management node receives computing tasks sent by the application system and determines computing information based on the computing tasks.

[0008] Based on the computing task and the resource information of each computing node, the main management node determines a first target computing node from among the multiple computing nodes and sends the computing information to the first target computing node.

[0009] According to the computing task, initial data is obtained from multiple computing nodes and / or multiple data sources through the first target computing node, and the computing task is completed based on the initial data.

[0010] In the above embodiments, the unified data processor consists of multiple computing nodes and a master management node. The computing nodes synchronously store data from multiple data sources (e.g., business databases and log file servers), while the master management node connects to the application system via a unified interface. The application system only needs to issue computing task instructions to the unified data processor, and the master management node and multiple computing nodes in the unified data processor can then perform corresponding data processing according to the computing task instructions to complete the computing task. The application system only needs to issue computing task instructions to the unified data processor without performing data processing, thereby reducing the data processing pressure on the application system. Furthermore, since the multiple computing nodes synchronously store data from multiple data sources, when the data involved in the computing task instructions is stored across multiple computing nodes, the first target computing node used for computing task processing can directly obtain the required initial data from multiple computing nodes, thereby reducing access to data sources during data processing, improving data processing speed, and reducing the impact on business database data storage operations.

[0011] In one possible implementation of the first aspect, the computing task includes N computing subtasks, and the computing task information also includes the association relationship of the N computing subtasks, each computing subtask corresponding to at least one first target computing node, where N is a positive integer greater than 1;

[0012] Based on the computing task and the resource information of each computing node, the first target computing node is determined from the plurality of computing nodes through the main management node, including:

[0013] Based on the N computing subtasks, the association relationships, and the resource information of each computing node, the first target computing node corresponding to each computing subtask is determined from the multiple computing nodes by the main management node.

[0014] In one possible implementation of the first aspect, the association includes the execution order of the N computational subtasks;

[0015] The main management node determines the first target computing node corresponding to each computing subtask from among the multiple computing nodes based on the N computing subtasks, the association relationships, and the resource information of each computing node, including:

[0016] Based on the N computational subtasks, the association relationships, and the resource information of each computational node, the main management node determines the first target computational node corresponding to each computational subtask from among the multiple computational nodes, including:

[0017] Based on the N computational subtasks and the associated relationships, the computational cost corresponding to the computational task is determined through the main management node;

[0018] If the computational cost is greater than a preset threshold, the main management node determines the corresponding first target computational node for each computational subtask from among the multiple computational nodes in the execution order.

[0019] When determining the first target computing node corresponding to the i-th computing subtask among the N computing subtasks, the first target computing node corresponding to the i-th computing subtask is determined from multiple computing nodes by the main management node based on the i-th computing subtask and the resource information of each computing node currently stored by the main management node, where i is a positive integer greater than 2.

[0020] The resource information of each computing node currently stored in the main management node is obtained by updating the resource information of each computing node stored in the main management node after the first target computing node corresponding to the (i-1)th computing subtask has completed the (i-1)th computing subtask.

[0021] After the first target computing node corresponding to the i-th computing subtask completes the i-th computing subtask, the resource information of each computing node currently stored is updated through the main management node. In one possible implementation of the first aspect, the unified data processor further includes multiple synchronization components, and the main management node stores the number of tasks for each synchronization component; the method further includes:

[0022] Obtain synchronization task instructions through the main management node;

[0023] Based on the number of tasks in each of the synchronization components, the synchronization task instruction is sent to the target synchronization component among the multiple synchronization components through the main management node;

[0024] According to the synchronization task instruction, the synchronization task information is determined by the target synchronization component;

[0025] Based on the synchronization task information and the resource information of each computing node, the second target computing node is determined from the multiple computing nodes by the target synchronization component, and the synchronization task information is sent to the second target computing node.

[0026] Based on the synchronization task information, the second target computing node establishes a connection with the target data source among the multiple data sources, and obtains and stores the data to be synchronized from the target data source.

[0027] In one possible implementation of the first aspect, the method further includes:

[0028] Obtain management task instructions through the main management node;

[0029] According to the management task instructions, the management task information is determined through the main management node;

[0030] Based on the management task information, the management task is executed through the main management node.

[0031] In one possible implementation of the first aspect, the method further includes:

[0032] Each computing node sends its own resource information to the main management node.

[0033] Based on the received resource information of each computing node, the main management node updates the stored resource information of each computing node.

[0034] In one possible implementation of the first aspect, at least one management node further includes at least one other management node, and the method further includes:

[0035] The resource information stored in the main management node is synchronized through the other management nodes.

[0036] Secondly, embodiments of this application provide a unified data processor, including multiple computing nodes and at least one management node, wherein the at least one management node includes a master management node, which is connected to an application system through a unified interface; wherein the master management node stores resource information of each computing node, and the multiple computing nodes synchronously store data from multiple data sources, wherein:

[0037] The main management node is used to receive computing task instructions sent by the application system and determine computing task information according to the computing task instructions;

[0038] The main management node is used to determine a first target computing node from multiple computing nodes based on the computing task information and the resource information of each computing node, and to send the computing task information to the first target computing node.

[0039] The first target computing node is used to obtain initial data from multiple computing nodes and / or multiple data sources according to the computing task information, and to perform calculations based on the initial data to complete the computing task corresponding to the computing task information.

[0040] In one possible implementation of the second aspect, the unified data processor further includes multiple synchronization components, and the master management node stores the number of tasks for each synchronization component; wherein:

[0041] The main management node is used to obtain synchronization task instructions;

[0042] The main management node is used to send the synchronization task instruction to the target synchronization component among the multiple synchronization components according to the number of tasks of each synchronization component;

[0043] The target synchronization component is used to determine synchronization task information according to the synchronization task instruction;

[0044] The target synchronization component is used to determine a second target computing node from multiple computing nodes based on the synchronization task information and the resource information of each computing node, and to send the synchronization task information to the second target computing node.

[0045] The second target computing node is used to establish a connection with the target data source among the multiple data sources according to the synchronization task information, and to obtain and store the data to be synchronized from the target data source.

[0046] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps performed by the management node or the computing node in any of the methods described in the first aspect above.

[0047] Fourthly, embodiments of this application provide a computer program product that, when run on a server, causes the server to execute the steps performed by the management node or the computing node in any of the methods described in the first aspect above.

[0048] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the unified data processor provided in one embodiment of this application;

[0051] Figure 2 This is an interactive diagram of the processing of computation task instructions in a data processing method provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart of a method for determining the first target computing node corresponding to each computing subtask, provided in another embodiment of this application;

[0053] Figure 4 This is an interaction diagram of the processing of a synchronization task in a data processing method provided in another embodiment of this application;

[0054] Figure 5 This is an interactive diagram illustrating the processing of management tasks in a data processing method provided in another embodiment of this application;

[0055] Figure 6 This is a structural block diagram of a data processing apparatus provided in an embodiment of this application;

[0056] Figure 7 This is a structural block diagram of a data processing apparatus provided in another embodiment of this application;

[0057] Figure 8 This is a structural block diagram of a data processing apparatus provided in another embodiment of this application;

[0058] Figure 9 This is a schematic diagram of the internal structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0059] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0060] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0061] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0062] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0063] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0064] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0065] Current methods for analyzing robot data involve application systems or clients directly querying and retrieving data from a business database, and then using this data for analysis and computation. Because data needs to be pulled from the business database over the network before computation, data processing efficiency is low, and the data storage of the business database is also affected by the data retrieval operation. When large amounts of robot data are needed for analysis and computation, the application system or client faces significant data processing pressure, and may even experience system crashes, making it difficult to complete the analysis and computation.

[0066] Therefore, this application provides a unified data processor, exemplarily, such as... Figure 1 The diagram shown is a structural schematic of a unified data processor provided in an embodiment of this application. The unified data processor includes multiple computing nodes and at least one management node. Figure 1 In this embodiment, at least one management node includes a main management node and other management nodes. The main management node is connected to the application system through a unified interface. The main management node stores and updates the resource information of each computing node. Multiple computing nodes synchronously store data from multiple data sources. The multiple computing nodes are interconnected through communication interfaces, and the data stored in each computing node can be transmitted to each other.

[0067] This application addresses the problems of low data processing efficiency caused by fetching data from the business database over the network before calculation, the impact of data retrieval on the business database, and the heavy data processing pressure on application systems or clients, making it difficult to analyze and calculate large amounts of robot data.

[0068] This application provides a data processing method based on the aforementioned unified data processor. The method includes a main management node in the unified data processor receiving a computing task instruction sent by an application system and parsing the instruction to obtain computing task information. The main management node and multiple computing nodes in the unified data processor perform corresponding data processing according to the computing task information to complete the computing task. The application system only needs to issue computing task instructions to the unified data processor without performing actual data processing, thus reducing the data processing pressure on the application system. Furthermore, since multiple computing nodes synchronously store data from multiple data sources, when the data involved in the computing task instruction is stored on multiple computing nodes, the unified data processor can directly obtain the initial data required for computing from multiple computing nodes, thereby reducing access to data sources during data processing, improving data processing efficiency, and reducing the impact on business database data storage operations.

[0069] like Figure 1 As shown, the unified data processor in this embodiment also includes multiple synchronization components, and the task quantity information of the multiple synchronization components is stored in the main management node. In this embodiment, the synchronization components are used to parse the synchronization task instructions issued by the main management node to obtain synchronization task information, and then send the synchronization task information to the determined second target computing node.

[0070] In this embodiment, each management node and each compute node can run on an independent server. Similarly, each synchronization component can also run on an independent server, or a synchronization component can run on the server where a management node resides or the service where a compute node resides. When a synchronization component runs on the server where a management node resides or the service where a compute node resides, the synchronization component and the management node or compute node on the same server are two independent programs, running independently without interfering with each other.

[0071] In this embodiment, there may be one or more management nodes, but only one management node can be the master management node at any given time. The master management node is responsible for the internal management of the unified data processor and its connection with external entities (such as application systems and clients). When the master management node fails, one of the other management nodes will become the master management node. Setting up multiple management nodes can improve the stability and security of the unified data processor.

[0072] For example, each management node includes a unified interface, metadata information, and task management services. The following is a detailed description:

[0073] In this embodiment, when the management node acts as the primary management node, its unified interface is responsible for interfacing with external systems, such as application systems and clients. Using a unified interface, that is, providing only a single, consistent connection protocol, shields the user from the complexity of data types from upstream application systems or clients.

[0074] In this embodiment, the metadata information in the management node can be the parameter information of each node (including management node, computing node, synchronization component, etc.) in the real-time database. The parameter information can specifically include: (1) resource information such as network information, CPU (central processing unit) information, memory information and disk information of each node (for computing nodes, the resource information also includes the data information of the synchronized data already stored in each computing point, such as the name, quantity, size, table fields, etc. of the synchronized database table); (2) task information in each node, such as synchronization tasks, computing tasks, etc.; (3) data statistics information of each node, etc.

[0075] Only when a management node is the primary management node can it directly update its internal metadata. When a management node is another management node, it updates its metadata by synchronizing with the metadata in the primary management node. Updating the metadata in the management node ensures that it reflects the latest information from all nodes in the unified data processor. This allows for the allocation of task information or instructions by combining the latest information from each node, thereby ensuring a balanced distribution of task information or instructions and guaranteeing the stability and efficiency of the unified data processor's data processing.

[0076] Understandably, the management tasks within a unified data processor include: updating metadata, receiving task instructions, distributing synchronization task instructions to synchronization components, parsing computation task instructions into computation task information and distributing them to computation nodes, parsing management task instructions into management task information and executing management tasks, and updating stored metadata. Updating stored metadata can be performed by the primary management node after completing its management tasks, or by the primary management node after receiving data reports from other nodes.

[0077] In this embodiment, the task management service is an application that performs specific operations in the management node, such as parsing task instructions, allocating task instructions or task information, updating or synchronizing metadata, etc.

[0078] Optionally, the task management service can also be used to monitor the progress of computing tasks in computing nodes. When a computing task fails or exits abnormally, the task management service is responsible for attempting to restart the process of the computing task in the computing node; when a computing node fails, it attempts to restart the process of the failed task on other computing nodes.

[0079] For example, a task failure or abnormal exit could be due to the inability to obtain the data corresponding to the computation task instruction, or the data source corresponding to the synchronization task instruction crashing and becoming unreachable. A compute node failure specifies that the server hosting the compute node executing the task is faulty, such as due to a power outage or other reasons.

[0080] In this embodiment, the computing nodes are responsible for executing specific tasks (e.g., computational tasks or synchronization tasks). Specifically, the computing nodes receive task information and execute tasks according to the task information. If the task is a computational task, the computing node executing the task obtains data from various computing nodes or data sources and performs the corresponding computations; if the task is a synchronization task, the computing node executing the task obtains the data that needs to be synchronized from the data source and stores it.

[0081] Understandably, there can be multiple compute nodes, each running on a server. Therefore, there can be multiple servers for compute nodes, and multiple servers providing compute node services simultaneously can effectively improve the computing power of real-time databases.

[0082] For example, each computing node includes an internal communication interface, a cross-source data service, a batch data synchronization service, a real-time data synchronization service, and a data storage service.

[0083] The internal communication interface is used for interconnection between various computing nodes, enabling the transmission of stored data, intermediate calculation results, and other data between the computing nodes.

[0084] Within a compute node, data computation services, cross-source data services, batch data synchronization services, real-time data synchronization services, and data storage services are all applications with different roles, each set up within the compute node. For ease of understanding, a detailed explanation follows:

[0085] In a compute node, the data compute service is set up for compute tasks. When a compute node receives a compute task, it starts the data compute service to execute the compute task.

[0086] In compute nodes, cross-source data services enable them to connect directly to external data sources. When a compute node is performing a computation task, if the required data is not stored on any of the compute nodes (data that has not yet been synchronized or does not need to be synchronized), the compute node can directly obtain the data from the external data source and perform computations through cross-source data services.

[0087] Within the compute nodes, batch data synchronization services and real-time data synchronization services are configured for synchronization tasks. These services are distinguished based on the different synchronization methods used in the tasks. The batch data synchronization service executes batch synchronization tasks, while the real-time data synchronization service executes real-time synchronization tasks. Optionally, each batch data synchronization service and each real-time data synchronization service can execute multiple corresponding data synchronization tasks to improve data synchronization capabilities.

[0088] Understandably, a real-time synchronization task refers to a synchronization task that needs to monitor for updates to the data files to be synchronized in real time. Whenever new data is added to the data files to be synchronized, the real-time data synchronization service is triggered to pull the new data from the data source. A batch synchronization task, on the other hand, refers to a synchronization task that determines the synchronization frequency upon receiving the task, and then performs synchronization according to the set frequency; for example, the synchronization frequency is once at a preset interval, or it can be a scheduled synchronization (such as synchronizing at a fixed time every day at midnight).

[0089] For example, whether a data synchronization task uses real-time synchronization or batch synchronization depends primarily on the real-time requirements of the data. Data with low real-time requirements is suitable for batch synchronization, such as data on the daily running time and mileage of robots. The application system only focuses on the final statistical data comparison each day, and such log details use batch synchronization. Data with high real-time requirements is suitable for real-time synchronization. For example, if the owner of a robot changes, the application system wants to quickly check the robot's ownership. Therefore, this type of robot ownership data needs to be synchronized in real-time.

[0090] Within the compute nodes, the data storage service performs specific storage processing on data acquired by the real-time data synchronization service and the batch data synchronization service, as well as handles read operations on the stored data. Specifically, during data storage, the data storage service encodes, indexes, and compresses the written data before storing it on the local disk. Optionally, the same data can exist in multiple replicas across different compute nodes to ensure high availability.

[0091] Understandably, when a synchronization task involves updating the same data, the data storage service will add a version number to the data before writing it directly, and will read the latest version of the data when reading it. In this embodiment, the data storage service will periodically merge historical data and delete older versions. The data storage service can automatically maintain the data storage of the entire unified data processor, thereby achieving efficient data writing and reading.

[0092] In a unified data processor, the synchronization component is configured for synchronization tasks, and it only participates in the data processing related to the synchronization task instructions. The synchronization component is illustrated below.

[0093] The system comprises multiple synchronization components, and their parameter information is stored in the metadata of the management node. This parameter information may include task details for each synchronization component, such as the number of tasks and their status. The management node distributes synchronization task instructions among the multiple synchronization components based on the metadata information. Each synchronization component reports its own task status information to the main management node.

[0094] Optionally, as mentioned above, different data may employ different data synchronization methods. Each data synchronization method corresponds to at least one synchronization component, and the same synchronization component can only correspond to one data synchronization method. Optionally, to ensure the high availability of the unified data processor, each data synchronization method may correspond to multiple synchronization components.

[0095] Understandably, for batch synchronization task commands, the corresponding synchronization component is a batch synchronization component. This component is solely used to interface with batch synchronization task commands, and during data synchronization, it corresponds to the batch data synchronization service within the compute node. Similarly, for real-time synchronization task commands, the corresponding synchronization component is a real-time synchronization component. This component is solely used to interface with real-time synchronization task commands, and during data synchronization, it corresponds to the real-time data synchronization service within the compute node.

[0096] The following describes a data processing method provided in this application by way of specific embodiments. The data processing method is applied to... Figure 1 On the unified data processor shown.

[0097] Figure 2 This diagram illustrates the processing interaction of computational task instructions in a data processing method provided in one embodiment of this application. As an example and not a limitation, the data processing method includes the following steps:

[0098] Step S201: The main management node receives the computing task instruction sent by the application system and determines the computing task information according to the computing task instruction.

[0099] In this embodiment, the main management node connects to the application system through a unified interface, and then receives computation task instructions sent by the application system through the unified interface. The application system can be any client with robot data analysis needs.

[0100] The unified interface is the only interface between the unified data processor and the outside world. That is, the unified data processor provides only one unified connection protocol to the outside world, thereby shielding the complexity of upstream data types.

[0101] Optionally, the computation task instructions can be SQL (Structured Query Language) commands. These instructions include the data tables and computational logic required to complete the task. Specifically, determining the computation task information based on the instructions involves parsing the SQL statements to generate the required information.

[0102] Specifically, the process of parsing computation task instructions may include: determining the required data files (e.g., data tables), the location of the required data files (each computation node or data source), and the size of the data volume based on the computation task instructions; then, based on the obtained data and the computation logic in the computation task instructions, determining the computation task information. The computation task information includes the computation task itself, which is an executable program file.

[0103] Step S202: The main management node determines the first target computing node from multiple computing nodes based on the computing task information and the resource information of each computing node, and sends the computing task information to the first target computing node.

[0104] In this embodiment, the most basic principle for determining the first target computing node is that the computing node has sufficient computing resources to undertake the corresponding computing task; wherein the first target computing node may include one or more, and the number of first target computing nodes is determined by the complexity of the computing task.

[0105] Optionally, the computation task may include N computational subtasks, where N is a positive integer. The computation task information includes the relationships between the N subtasks, meaning that the N computational subtasks are interconnected. These relationships may include the execution order of the N computational subtasks, the transmission relationship of intermediate computation results between the various computational subtasks, etc.

[0106] In one embodiment, the method for determining the first target computing node includes: the main management node determining the first target computing node corresponding to each computing subtask from multiple computing nodes based on N computing subtasks, their relationships, and the resource information of each computing node. Each computing subtask may correspond to one or more first target computing nodes, and the same computing node may serve as the first target computing node for one or more computing subtasks.

[0107] In this embodiment, the computational subtask is obtained by breaking down the computational task into specific steps. For ease of understanding, the acquisition of the computational subtask is illustrated with an example. For instance, suppose the computational task is: to count the daily mileage of all robots in each store and sort the mileage of robots in each store. This computational task requires the operation log data of each robot, as well as the relationship data between each robot and its respective store.

[0108] The calculation task is broken down into the following subtasks: data reading, mileage calculation, store attribution, and mileage sorting. These subtasks are interconnected. The specific relationships are as follows:

[0109] (1) Data reading subtask: Responsible for reading robot log data and robot-bound store relationship data, and after completing data reading, transmitting the data to the mileage calculation subtask;

[0110] (2) Mileage Calculation Subtask: Responsible for calculating the mileage of each robot and then sending the mileage statistics of each robot to the store's assigned subtask.

[0111] (3) Store Assignment Subtask: Responsible for assigning each robot to a store based on the table data of the machine-bound stores, and sending the assigned data to the Mileage Sorting Subtask.

[0112] (4) Mileage sorting subtask: responsible for sorting the robot mileage in each store based on the received data.

[0113] After identifying the different computational subtasks, the main management node determines the first target computational node to execute each computational subtask based on the multiple computational subtasks, their relationships, and the resource information of each computational node.

[0114] Regarding the determination of the primary target computing node for different computational subtasks, assuming sufficient computing resources (e.g., CPU, memory, and disk), the computing node containing the data required by the subtask is preferentially selected as the primary target computing node to minimize data transfer workload. Sufficient computing resources mean that the computing resources of the computing node are sufficient to complete the corresponding computational subtask. Whether the computing resources of the computing node are sufficient can be determined based on the node's resource information and the number of computational subtasks. Optionally, the amount of computational data can be characterized by the size of the memory occupied by the data during the execution of the computational subtask; the larger the memory occupied by the data, the larger the amount of computational data.

[0115] Step S203: The first target computing node obtains initial data from multiple computing nodes and / or multiple data sources based on the computing task information, and performs calculations based on the initial data to complete the computing task corresponding to the computing task information.

[0116] In this embodiment, after determining the first target computing node, the first target computing node executes a computing task. The initial data required for executing the computing task may or may not be synchronized to the computing node; the first target computing node preferentially obtains the initial data from the computing node, and if the initial data is not synchronized to the computing node, the first target computing node obtains the initial data from the data source.

[0117] Optionally, the data processing method also includes: each computing node sending its own resource information to the master management node; and the master management node updating the stored resource information of each computing node based on the received resource information. This method, which involves each computing node reporting its own resource information, ensures that the resource information of each computing node in the master management node is updated. This approach is simple and easy to implement.

[0118] Optionally, at least one management node may also include at least one other management node, and the method further includes: each other management node synchronizing the resource information of each computing node stored in the main management node. By setting up other management nodes and synchronizing the resource information of each computing node in the main management node, when the main management node fails, it is convenient for other management nodes to become the main management node and perform the corresponding functions.

[0119] In one embodiment, a one-time orchestration method can be used to determine the first target computing node corresponding to each computing subtask. That is, before task execution, the first target computing node corresponding to each computing subtask is determined based on the resource information of each computing node and the status of the computing subtasks. Then, each computing subtask is executed sequentially using the determined first target computing node. The one-time orchestration method is simple, has a fast orchestration speed, and executes each computing subtask after orchestration, resulting in high processing efficiency.

[0120] The following example, using the determination of the first target calculation node for the data reading subtask, mileage calculation subtask, store affiliation subtask, and mileage sorting subtask in the above example, illustrates the process of one-time orchestration for each calculation subtask. Assume that calculation node A stores robot operation log data, and calculation node B stores table data of robot-bound stores. The orchestration process includes:

[0121] (1) First, based on the resource information of computing nodes A and B, determine whether the computing resources of computing nodes A and B are sufficient to execute the data reading subtask. Since computing nodes A and B have sufficient computing resources, they are selected as the first target computing nodes for the data reading subtask.

[0122] (2) Estimate whether computing node A has sufficient computing resources to execute the mileage calculation subtask after obtaining the robot operation log data. After estimation, it was found that computing node A has sufficient computing resources. Therefore, computing node A was selected as the first target computing node for the mileage calculation subtask.

[0123] (3) Estimating whether computing nodes A and B have sufficient computing resources to execute the store-assigned subtask, the estimation method found that both computing nodes A and B have sufficient computing resources. Given that both computing nodes A and B have sufficient computing resources, a comparison was made between the predicted mileage statistics data volume and the predicted data volume of the robot-bound store table data. The estimation results showed that the predicted mileage statistics data volume would be much smaller than the predicted data volume of the robot-bound store table data. To reduce the time spent on data transmission, computing node B was selected as the first target computing node for the store-assigned subtask.

[0124] (4) Estimate whether computing node B has sufficient computing resources to handle the mileage sorting subtask. After estimation, computing node B has sufficient computing resources, so computing node B is selected as the first target computing node for the mileage sorting subtask.

[0125] After the above one-time arrangement, computing node A and computing node B are both the first target computing nodes corresponding to the data reading subtask; computing node A is the first target computing node for the mileage calculation subtask; computing node B is the first target computing node for the store affiliation subtask; and computing node B is the first target computing node for the mileage sorting subtask.

[0126] The specific process for executing the above computational subtasks includes:

[0127] (1) The main management node sends the data reading subtask to computing node A and computing node B respectively, sends the mileage calculation subtask to computing node A, and sends the store affiliation subtask and mileage sorting subtask to computing node B;

[0128] (2) Computing node A executes the data reading subtask to read robot operation log data from the local storage data; computing node B executes the data reading subtask to read table data of the robot's bound stores from the local storage data.

[0129] (3) Computing node A executes the mileage calculation subtask, obtains the mileage data statistics results of each robot based on the robot operation log data, and sends the mileage data statistics results to computing node B.

[0130] (4) Calculation node B executes the store-assigned subtask to match the robot's mileage data statistics results to each store.

[0131] (5) Calculation node B executes the mileage sorting subtask to sort the robot mileage data for each store.

[0132] In the above embodiments, when computing resources are sufficient, the corresponding computational subtasks are executed on the computing nodes where the data resides, thereby reducing data transfer between computing nodes and improving processing efficiency. Furthermore, when executing a computational subtask requires data from multiple computing nodes, when computing resources are sufficient, smaller data volumes are sent to the computing nodes containing larger data volumes, allowing the larger computing nodes to process the corresponding computational subtasks; this shortens the time spent on data transfer and improves processing efficiency.

[0133] In one possible scenario, to better ensure that each computational subtask selects a first target computing node with sufficient computing resources, a dynamic orchestration method can be used to determine the first target computing node for each computational subtask. The association includes the execution order of the N computational subtasks. The dynamic orchestration method mainly includes: the main management node, according to the execution order, sequentially determining the corresponding first target computing node for each computational subtask from multiple computing nodes; specifically, when determining the corresponding first target computing node for the i-th computational subtask among the N computational subtasks, the main management node determines the first target computing node corresponding to the i-th computational subtask from multiple computing nodes based on the i-th computational subtask and the resource information of each computing node currently stored by the main management node.

[0134] The resource information of each computing node currently stored in the master management node is obtained by updating the resource information of each computing node stored in the master management node after the first target computing node corresponding to the (i-1)th computing subtask has completed the (i-1)th computing subtask.

[0135] After the first target computing node corresponding to the i-th computing subtask completes the execution of the i-th computing subtask, the main management node updates the resource information of each computing node currently stored.

[0136] The dynamic orchestration method described above determines the first target computing node for each computing subtask as it is executed. After completing a computing subtask, the computing resource information of each computing node is updated. When determining the first target computing node for the next computing subtask, the latest computing resource information of each computing node can be used to ensure that the determined first target computing node has sufficient computing resources to complete the corresponding computing subtask, thereby better ensuring that each computing subtask can be completed smoothly.

[0137] Figure 3 This is a flowchart of a method for determining the first target computing node corresponding to each computing subtask in one embodiment. Figure 3 The method shown selects between one-time orchestration and dynamic orchestration based on the specific circumstances of the computational task, thus combining the simplicity and convenience of one-time orchestration with the stability of dynamic orchestration. For example... Figure 3 As shown, the method includes:

[0138] Step S301: The main management node determines the computation cost corresponding to the computation task based on the N computation sub-tasks and their relationships.

[0139] In this embodiment, the computational cost refers to the time required to complete the corresponding computational task. In this embodiment, the computational cost is estimated based on each computational subtask and its relationship.

[0140] Understandably, the computational cost of a computational task is determined by its computational complexity and the size of the data being processed. The higher the computational complexity and the larger the amount of data, the greater the computational cost.

[0141] The computational cost is directly proportional to the computational complexity, which is determined by the operators in the computational subtasks. Different types of operators correspond to different computational complexities. Therefore, preset values ​​for operator costs (preset values ​​for the computational costs corresponding to the operators) can be pre-marked for the same amount of data according to the operator type. For example, for the same amount of data, the computation time of the sorting operator is twice that of the summation operator. Therefore, the preset value for the complex cost of the sorting operator can be set to 2, and the preset value for the complex cost of the summation operator can be set to 1.

[0142] Since the various computational subtasks have not yet been executed, it is impossible to obtain the actual computational data volume of the other computational subtasks besides the first one. However, an approximate estimate can be made based on the operator type and execution order. For example, if the operators in the previous computational subtask have performed data aggregation, then the data volume will decrease, meaning the influence coefficient of the previous computational subtask on the operator data volume of the current computational subtask is less than 1. Conversely, if the number of operators in the previous computational subtask has increased, the influence coefficient of the previous computational subtask on the operator data volume of the current computational subtask is greater than 1.

[0143] For example, the computational cost of a computational task can be estimated using the following formula:

[0144] T = a * (Operator 1 cost preset value + Operator 1 cost preset value * Operator 1 data volume influence coefficient + Operator 3 cost preset value * Operator 1 data volume influence coefficient * Operator 2 data volume influence coefficient + ... + Operator n cost preset value * Operator 1 data volume influence coefficient * Operator 2 data volume influence coefficient * ... * Operator (n-1) data volume influence coefficient)

[0145] Where: * represents the multiplication symbol;

[0146] T represents the estimated computational cost of the computational task;

[0147] 'a' represents the amount of data to be labeled, where the amount of initial data read by the computation task is a GB.

[0148] Operator 1 cost preset value, which represents the cost preset value of the operator corresponding to the first computational subtask;

[0149] The preset cost value for operator 2 represents the preset cost value of the operator corresponding to the second computational subtask.

[0150] ...

[0151] The preset cost value for operator n represents the preset cost value for the operator corresponding to the second computational subtask;

[0152] The data volume influence coefficient of operator 1 represents the influence coefficient of the first computational subtask on the data volume;

[0153] The data volume influence coefficient of operator 2 represents the influence coefficient of the second computational subtask on the data volume;

[0154] ...

[0155] The (n-1) data volume influence coefficient of the operator represents the influence coefficient of the (n-1)th computational subtask on the data volume.

[0156] Step S302: If the computation cost is greater than the preset threshold, the main management node determines the first target computation node corresponding to each computation subtask according to the dynamic arrangement method.

[0157] In this embodiment, when the computation cost is greater than a preset threshold, it indicates that it takes a long time to complete all computation subtasks. Therefore, a dynamic orchestration method is used to determine the first target computing node corresponding to each computation subtask, so as to avoid the situation where the computing resource status of the first target computing node determined by the one-time orchestration changes due to the long computation time of the computation task, and the corresponding computation subtask cannot be executed.

[0158] Step S303: If the computation cost is less than the preset threshold, the main management node determines the first target computation node corresponding to each computation subtask according to the one-time arrangement method.

[0159] In this embodiment, for computational tasks with a computational cost less than a preset threshold, the first target computational node corresponding to each computational subtask can be determined by a one-time arrangement method.

[0160] Figure 4 This is a process interaction diagram of the synchronization task in a data processing method provided in one embodiment of this application, such as... Figure 4 As shown, the processing procedure for a synchronization task specifically includes:

[0161] Step S401: The main management node obtains the synchronization task instruction.

[0162] In this embodiment, the synchronization task instruction can be issued by the application system to the main management node, or it can be issued by the background client to the main management node.

[0163] Optionally, synchronization task commands can be SQL commands, and the specific content included in the synchronization task commands differs depending on the synchronization method. Additionally, different task types have different task type keywords, which can be used to distinguish the synchronization method of the synchronization task commands.

[0164] Generally, batch synchronization task instructions include the connection method of the data source (e.g., business database address, business database username and password), the databases / tables / fields to be synchronized, the databases / tables to be written to in the unified data processor, and the batch synchronization task time information (start and end times of the synchronization period, and the frequency of synchronization task execution). Real-time synchronization task instructions differ from batch synchronization task instructions in that they include, but are not limited to, the lifecycle of the batch synchronization task; other parts are the same.

[0165] Step S402: The main management node sends the synchronization task instruction to the target synchronization component among the multiple synchronization components according to the number of tasks of each synchronization component.

[0166] In this embodiment, the main management node stores task information (such as the number of tasks and task status) of each synchronization component. The main management node determines the target synchronization component from multiple synchronization components based on the number of tasks of each synchronization task component. When determining the target synchronization component, the main management node mainly distributes the synchronization task instructions evenly to each synchronization task component, thereby making full use of the resources of each synchronization task component and improving data processing efficiency.

[0167] In this embodiment, different synchronization components are set for different synchronization methods. That is, the multiple synchronization components include at least one real-time synchronization component and one batch synchronization component. Optionally, there can be multiple real-time synchronization components, and there can also be multiple batch synchronization components. The real-time synchronization component is only used to receive and process real-time synchronization task instructions, and the batch synchronization component is only used to receive and process batch synchronization task instructions.

[0168] Step S403: The target synchronization component determines the synchronization task information according to the synchronization task instruction.

[0169] In this embodiment, the target synchronization component parses the synchronization task instructions to obtain synchronization task information. The synchronization task information includes the synchronization task itself, which is an executable program file.

[0170] In this embodiment, during the parsing of synchronization task instructions, the instructions need to be validated. Validation includes: whether the information in the synchronization task instruction is complete, and whether the test data source is reachable. If the synchronization task instruction is a batch synchronization task instruction, the validation also includes: verifying whether the batch synchronization task time information is correct. When all validations pass, the synchronization task information is obtained based on the synchronization task instruction.

[0171] The main requirements for verifying the correctness of the time information of batch synchronization tasks are: the end time of the synchronization period is greater than the start time, the time format is correct (e.g., including year, month, day, hour, minute, second, etc.), and the execution frequency is executable (e.g., the execution frequency is a positive integer).

[0172] Step S404: The target synchronization component determines the second target computing node from multiple computing nodes based on the synchronization task information and the resource information of each computing node, and sends the synchronization task information to the second target computing node.

[0173] In this embodiment, the most basic principle for determining the second target computing node is that the node has sufficient computing resources to undertake the corresponding synchronization task. Computing nodes with more idle computing resources are preferentially selected as the second target computing node.

[0174] Optionally, upon receiving the synchronization task information, the second target computing node initiates the corresponding data synchronization service. For example, if the synchronization task information is a real-time synchronization task, the second target table computing node initiates the real-time data synchronization service to execute the real-time synchronization task; if the synchronization task information is a batch synchronization task, the second target table computing node initiates the batch data synchronization service to execute the batch synchronization task.

[0175] Step S405: The second target computing node establishes a connection with the target data source among multiple data sources based on the synchronization task information, and obtains and stores the data that needs to be synchronized from the target data source.

[0176] In this embodiment, when the second target computing node executes the corresponding synchronization task, it specifically includes: the second target computing node first establishes a connection with the target data source according to the connection method of the data source in the synchronization task information; then, it obtains the data to be synchronized from the target data source according to the information such as the database / table / field to be synchronized in the synchronization task information; and finally, it stores the obtained data to be synchronized.

[0177] Specifically, if the synchronization task is a real-time synchronization task, after the second target computing node establishes a connection with the target data source, it will monitor the target data source, obtaining the databases / tables / fields that need to be synchronized. Whenever new data is added to the target data source, it will retrieve and store the new data. If the synchronization task is a batch synchronization task, after the second target computing node establishes a connection with the target data source, it will synchronize and store the data that needs to be synchronized according to the synchronization frequency in the synchronization task information within the synchronization time period.

[0178] It is understandable that not all data in the data source needs to be synchronized. Generally, data that needs to be synchronized includes data whose volume exceeds the first threshold (i.e., large data volume) and data whose usage frequency exceeds the second threshold (i.e., frequently used data).

[0179] For large datasets, retrieving data from the data source over the network only when needed without synchronization requires high network stability and incurs significant time consumption, resulting in low data processing efficiency. Therefore, large datasets generally need to be synchronized to a unified data processor. Similarly, for frequently used data, failure to synchronize would necessitate frequent data retrieval from the data source over the network, further reducing processing efficiency. Therefore, frequently used data also needs to be synchronized to a unified data processor.

[0180] Figure 5 The flowchart illustrates the processing procedure of a management task in a data processing method provided in an embodiment of this application. The processing procedure of the management task specifically includes:

[0181] Step S501: The main management node obtains management task instructions.

[0182] In this embodiment, management task instructions are typically issued by the system administrator through a unified interface connected to the unified data processor. For example, management instructions can be sent from a background client.

[0183] Step S502: The main management node determines the management task information according to the management task instruction.

[0184] Optionally, management task commands can be SQL commands. The parsing process for management task commands is similar to that for synchronization task commands, and will not be elaborated here.

[0185] Step S503: The main management node executes management tasks based on the management task information.

[0186] In this embodiment, the management task may include node operations and system operations. The node corresponding to the node operation may be a computing node, a management node, or a synchronization component. The node operation may be node deletion, node addition, node startup, or node shutdown. The system operation corresponds to the unified data processor, and the system operation may specifically be system startup or system shutdown.

[0187] Optionally, if the management task performed by the master management node is specifically a node operation targeting the compute nodes, then after the master management node completes the management task, the master management node updates the resource information of each compute node based on the node operation results.

[0188] In this embodiment, after performing node operations, the master management node updates the resource information of each computing node, specifically by updating the metadata in the master management node.

[0189] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0190] Corresponding to the steps performed by the management node in the data processing method described in the above embodiments, Figure 6 A structural block diagram of a data processing apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0191] Reference Figure 6 The data processing device 600 includes:

[0192] The receiving unit 601 is used to receive the computing task instruction sent by the application system and determine the computing task information according to the computing task instruction;

[0193] The first determining unit 602 is used to determine the first target computing node from multiple computing nodes based on computing task information and resource information of each computing node.

[0194] The sending unit 603 is used to send computing task information to the first target computing node, so that the first target computing node can obtain initial data from multiple computing nodes and / or multiple data sources according to the computing task information, and perform calculations based on the initial data to complete the computing task corresponding to the computing task information.

[0195] Optionally, the computing task includes N computing subtasks, and the computing task information also includes the association relationships of the N computing subtasks. Each computing subtask corresponds to at least one first target computing node, where N is a positive integer. The first determining unit 602 is used to determine the first target computing node from multiple computing nodes based on the computing task information and the resource information of each computing node, specifically including:

[0196] Based on the N computational subtasks, the association relationships, and the resource information of each computational node, the first target computational node corresponding to each computational subtask is determined from the plurality of computational nodes.

[0197] Optionally, the association relationship includes the execution order of the N computational subtasks; the first determining unit 602, when determining the first target computational node corresponding to each computational subtask from the plurality of computational nodes based on the N computational subtasks, the association relationship, and the resource information of each computational node, specifically includes:

[0198] Based on the N computational subtasks and the associated relationships, determine the computational cost corresponding to the computational task;

[0199] If the computational cost is greater than the preset threshold, the first target computational node corresponding to each computational subtask is determined sequentially from the plurality of computational nodes according to the execution order;

[0200] When determining the first target computing node for the i-th computing subtask among the N computing subtasks, the first determining unit 602 determines the first target computing node corresponding to the i-th computing subtask from among the multiple computing nodes based on the i-th computing subtask and the resource information of each computing node currently stored in the main management node.

[0201] The resource information of each computing node currently stored in the main management node is obtained by updating the resource information of each computing node stored in the main management node after the first target computing node corresponding to the (i-1)th computing subtask has completed the (i-1)th computing subtask.

[0202] After the first target computing node corresponding to the i-th computing subtask completes the execution of the i-th computing subtask, the main management node updates the resource information of each computing node currently stored.

[0203] Optionally, the unified data processor further includes multiple synchronization components, wherein:

[0204] The receiving unit 601 is also used to acquire synchronization task instructions;

[0205] The sending unit 603 is further configured to send the synchronization task instruction to a target synchronization component among the plurality of synchronization components according to the number of tasks of each of the synchronization components; so that the target synchronization component determines the synchronization task information according to the synchronization task instruction, and determines a second target computing node from the plurality of computing nodes according to the synchronization task information and the resource information of each of the computing nodes, and sends the synchronization task information to the second target computing node; thereby enabling the second target computing node to determine a target data source from the plurality of data sources according to the synchronization task information, and to obtain and store the data to be synchronized from the target data source.

[0206] Optionally, the data processing device 600 further includes a second determining unit and an execution unit, wherein:

[0207] The receiving unit 601 is also used to acquire management task instructions;

[0208] The second determining unit is used to determine management task information according to the management task instruction;

[0209] An execution unit is used to execute management tasks based on the management task information.

[0210] Optionally, the data processing device 600 further includes an update unit, wherein:

[0211] The receiving unit 601 is also used to receive the resource information of each of the computing nodes.

[0212] The update unit is used to update the stored resource information of each computing node based on the received resource information of each computing node.

[0213] Corresponding to the steps performed by the computing node in the data processing method described in the above embodiments, Figure 7 A structural block diagram of a data processing apparatus provided in another embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0214] Reference Figure 7 The data processing device 700 includes:

[0215] The receiving unit 701 is used to receive computing task information determined by the main management node according to the computing task instructions sent by the application system;

[0216] The processing unit 702 is configured to obtain initial data from multiple computing nodes and / or multiple data sources according to the computing task information, and perform calculations based on the initial data to complete the computing task corresponding to the computing task information.

[0217] Optionally, the receiving unit 701 is also used to receive synchronization task information sent by the target synchronization component, the synchronization task information being determined by the target synchronization component according to the synchronization task instruction sent by the master management node.

[0218] The processing unit 702 is further configured to determine a target data source from multiple data sources based on the synchronization task information, and to obtain and store the data to be synchronized from the target data source.

[0219] The data processing apparatus 700 further includes: a sending unit, configured to send its own resource information to the main management node, so that the main management node updates the stored resource information of each computing node based on the received resource information of each computing node.

[0220] Corresponding to the steps performed by the synchronization component in the data processing method described in the above embodiments, Figure 8 A structural block diagram of a data processing apparatus provided in another embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0221] Reference Figure 8 The data processing device 800 includes:

[0222] The receiving unit 801 is used to receive a synchronization task instruction sent by the master management node according to the number of tasks of each of the synchronization components, wherein the synchronization task instruction is obtained by the master management node.

[0223] Determining unit 802 is used to determine synchronization task information according to the synchronization task instruction;

[0224] The processing unit 803 is configured to determine a second target computing node from among the multiple computing nodes based on the synchronization task information and the resource information of each computing node, and send the synchronization task information to the second target computing node, so that the second target computing node determines a target data source from among the multiple data sources based on the synchronization task information, and obtains and stores the data to be synchronized from the target data source.

[0225] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0226] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0227] Based on the same inventive concept, one embodiment of this application also provides a data processing device. For example... Figure 9 As shown, the data processing device 900 of this embodiment includes a processor 901, a memory 902, and a computer program 904 stored in the memory 902 and executable on the processor 901. The computer program 904 can be executed by the processor 901 to generate instructions 903, which the processor 901 can use to implement the steps in the above-described intelligent device method embodiments. Alternatively, when the processor 901 executes the computer program 904, it implements the functions of each module / unit in the above-described device embodiments.

[0228] For example, computer program 904 may be divided into one or more modules / units, one or more of which are stored in memory 902 and executed by processor 901 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 904 in data processing device 900.

[0229] The processor 901 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0230] The memory 902 can be an internal storage unit of the data processing device 900, such as a hard disk or RAM of the data processing device 900. The memory 902 can also be an external storage device of the data processing device 900, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the data processing device 900. Furthermore, the memory 902 can include both internal and external storage units of the data processing device 900. The memory 902 is used to store computer programs and other programs and data required by the data processing device 900. The memory 902 can also be used to temporarily store data that has been output or will be output.

[0231] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps performed by the management node, the computing node, or the synchronization component in the above-described method embodiments.

[0232] This application provides a computer program product that, when run on a server, enables the server to execute the steps performed by the management node, the computing node, or the synchronization component as described in the above-described method embodiments.

[0233] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / server, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0234] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0235] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0237] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0238] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The unified data processor is applied to a unified data processor, which includes multiple computing nodes and at least one management node, wherein the at least one management node includes a master management node, and the master management node is connected to the application system through a unified interface. The master management node stores resource information for each computing node, and the multiple computing nodes synchronously store data from multiple data sources; the method includes: The main management node receives computing task instructions sent by the application system and determines computing task information based on the computing task instructions. Based on the computing task information and the resource information of each computing node, the main management node determines a first target computing node from among the multiple computing nodes and sends the computing task information to the first target computing node. According to the computing task information, initial data is obtained from multiple computing nodes and / or multiple data sources through the first target computing node, and the computing task corresponding to the computing task information is completed based on the initial data. The computing task includes N computing subtasks, and the computing task information also includes the association relationship of the N computing subtasks. Each computing subtask corresponds to at least one first target computing node, and N is a positive integer greater than 1. Based on the computing task and the resource information of each computing node, the first target computing node is determined from the plurality of computing nodes through the main management node, including: Based on the N computing subtasks, the association relationships, and the resource information of each computing node, the first target computing node corresponding to each computing subtask is determined from the multiple computing nodes by the main management node; The association includes the execution order of the N computational subtasks; Based on the N computational subtasks, the association relationships, and the resource information of each computational node, the main management node determines the first target computational node corresponding to each computational subtask from among the multiple computational nodes, including: Based on the N computational subtasks and the associated relationships, the computational cost corresponding to the computational task is determined through the main management node; If the computational cost is greater than a preset threshold, the main management node determines the corresponding first target computational node for each computational subtask from among the multiple computational nodes in the execution order. When determining the first target computing node corresponding to the i-th computing subtask among the N computing subtasks, the first target computing node corresponding to the i-th computing subtask is determined from multiple computing nodes by the main management node based on the i-th computing subtask and the resource information of each computing node currently stored by the main management node, where i is a positive integer greater than 2. The resource information of each computing node currently stored in the main management node is obtained by updating the resource information of each computing node stored in the main management node after the first target computing node corresponding to the (i-1)th computing subtask has completed the (i-1)th computing subtask. After the first target computing node corresponding to the i-th computing subtask completes the i-th computing subtask, the resource information of each computing node currently stored is updated through the main management node.

2. The method as described in claim 1, characterized in that, The unified data processor further includes multiple synchronization components, and the master management node stores the number of tasks for each synchronization component; the method further includes: Obtain synchronization task instructions through the main management node; Based on the number of tasks in each of the synchronization components, the synchronization task instruction is sent to the target synchronization component among the multiple synchronization components through the main management node; According to the synchronization task instruction, the synchronization task information is determined by the target synchronization component; Based on the synchronization task information and the resource information of each computing node, the second target computing node is determined from the multiple computing nodes by the target synchronization component, and the synchronization task information is sent to the second target computing node. Based on the synchronization task information, the second target computing node establishes a connection with the target data source among the multiple data sources, and obtains and stores the data to be synchronized from the target data source.

3. The method as described in claim 1, characterized in that, The method further includes: Obtain management task instructions through the main management node; According to the management task instructions, the management task information is determined through the main management node; Based on the management task information, the management task is executed through the main management node.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Each computing node sends its own resource information to the main management node. Based on the received resource information of each computing node, the main management node updates the stored resource information of each computing node.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes at least one other management node, and at least one management node is also included in the above method: The resource information stored in the main management node is synchronized through the other management nodes.

6. A unified data processor, characterized in that, It includes multiple computing nodes and at least one management node, wherein the at least one management node includes a master management node, and the master management node is connected to the application system through a unified interface; The master management node stores resource information for each computing node, and the multiple computing nodes synchronously store data from multiple data sources, wherein: The main management node is used to receive computing task instructions sent by the application system and determine computing task information according to the computing task instructions; The main management node is used to determine a first target computing node from multiple computing nodes based on the computing task information and the resource information of each computing node, and to send the computing task information to the first target computing node. The first target computing node is used to obtain initial data from multiple computing nodes and / or multiple data sources according to the computing task information, and to perform calculations based on the initial data to complete the computing task corresponding to the computing task information; The computing task includes N computing subtasks, and the computing task information also includes the association relationship of the N computing subtasks. Each computing subtask corresponds to at least one first target computing node, and N is a positive integer greater than 1. The master management node is used to determine a first target computing node from a plurality of computing nodes based on the computing task and the resource information of each computing node, including: The main management node is used to determine the first target computing node corresponding to each computing subtask from multiple computing nodes based on the N computing subtasks, the association relationship and the resource information of each computing node; The association includes the execution order of the N computational subtasks; The main management node is used to determine the first target computing node corresponding to each computing subtask from among multiple computing nodes based on the N computing subtasks, the association relationships, and the resource information of each computing node, including: The main management node is used to determine the computational cost corresponding to the computational task based on the N computational subtasks and the association relationship; If the computational cost is greater than a preset threshold, the main management node is used to determine the corresponding first target computing node for each computing subtask from among the multiple computing nodes in the execution order. When determining the first target computing node corresponding to the i-th computing subtask among the N computing subtasks, the main management node is used to determine the first target computing node corresponding to the i-th computing subtask from multiple computing nodes based on the i-th computing subtask and the resource information of each computing node currently stored by the main management node, where i is a positive integer greater than 2. The resource information of each computing node currently stored in the main management node is obtained by updating the resource information of each computing node stored in the main management node after the first target computing node corresponding to the (i-1)th computing subtask has completed the (i-1)th computing subtask. After the first target computing node corresponding to the i-th computing subtask completes the i-th computing subtask, the main management node updates the resource information of each computing node currently stored.

7. The unified data processor as described in claim 6, characterized in that, It also includes multiple synchronization components, and the main management node stores the number of tasks for each synchronization component; wherein: The main management node is used to obtain synchronization task instructions; The main management node is used to send the synchronization task instruction to the target synchronization component among the multiple synchronization components according to the number of tasks of each synchronization component; The target synchronization component is used to determine synchronization task information according to the synchronization task instruction; The target synchronization component is used to determine a second target computing node from multiple computing nodes based on the synchronization task information and the resource information of each computing node, and to send the synchronization task information to the second target computing node. The second target computing node is used to establish a connection with the target data source among the multiple data sources according to the synchronization task information, and to obtain and store the data to be synchronized from the target data source.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps performed by the management node or the steps performed by the computing node in the method as described in any one of claims 1 to 5.

Images