Distributed data processing method, device and equipment

Abstract

The invention discloses a distributed data processing method and device, electronic equipment and a storage medium, and the method comprises the steps that a distributed system comprises a server and a plurality of computing nodes, and the server obtains a data processing task of to-be-processed data; the to-be-processed data is stored in a plurality of computing nodes in a distributed manner; dividing the data processing task into a plurality of sub-tasks, and determining a task node corresponding to each sub-task from the computing nodes; distributing the sub-tasks to corresponding task nodes to enable the task nodes to obtain to-be-processed data corresponding to the sub-tasks from the computing nodes, and returning sub-task processing results obtained by processing the to-be-processed data corresponding to the sub-tasks to the server; and aggregating the received sub-task processing results to obtain a target processing result. Therefore, large-scale to-be-processed data can be processed, the computing power of each computing node can be fully utilized, and the computing efficiency is relatively high.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and specifically relates to a distributed data processing method, apparatus, device and storage medium. Background Art

[0002] As data becomes a new dimension of means of production, its importance is increasing. How to conduct fast and efficient data analysis applications has become an important research direction for digital support. It can support the monitoring of various business indicators, timely understand the competitive situation and market share, help predict future business development trends, and provide important reference for future strategic planning and decision-making.

[0003] Among them, spreadsheet processing software is one of the most frequently used software in daily office work. It can provide rich functions such as data sorting, filtering, perspective, and chart generation, and simple analysis and processing of data can be completed with simple mouse operations.

[0004] However, in the prior art, there is a certain limit on the number of rows of data that can be loaded by spreadsheet processing software, and data sets exceeding the preset number of rows cannot be loaded. Therefore, large-scale data visualization analysis cannot be supported, and large-scale data sets cannot be processed. Moreover, spreadsheet processing software is usually run on a single machine. Due to the processing power of a single machine, when processing data sets with a slightly larger data scale, abnormal situations such as operation jams and slow responses are prone to occur. These problems seriously hinder the application of spreadsheet processing software. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a distributed data processing method, apparatus, device and storage medium, which can solve the problem that current electronic spreadsheet processing software cannot process large-scale data sets and is limited by the processing power of a single machine, and is prone to abnormal situations such as operation jams and slow responses.

[0006] In a first aspect, an embodiment of the present application provides a distributed data processing method, which is applied to a server in a distributed system, wherein the distributed system also includes multiple computing nodes, and the method includes:

[0007] Acquire a data processing task for the data to be processed; the data to be processed is distributed and stored in a plurality of the computing nodes;

[0008] Divide the data processing task into a plurality of subtasks, and determine a task node corresponding to each subtask from the computing nodes;

[0009] Allocating the subtask to a corresponding task node so that the task node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server;

[0010] The received subtask processing results are aggregated to obtain the target processing result.

[0011] Optionally, the acquiring of a data processing task for the data to be processed includes:

[0012] Get the data processing request sent by the user;

[0013] Parsing the data processing request to obtain a data processing task for the data to be processed;

[0014] After aggregating the received subtask processing results to obtain the target processing results, the method further includes:

[0015] The target processing result is returned to the user terminal for display.

[0016] Optionally, before obtaining the data processing task for the data to be processed, the method includes:

[0017] Get the data to be processed;

[0018] Divide the data to be processed into multiple subsets, determine at least one storage node corresponding to each subset from the computing nodes, and allocate the subsets to the corresponding storage nodes respectively;

[0019] The task node obtains the to-be-processed data corresponding to the subtask from any of the storage nodes, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server.

[0020] Optionally, dividing the data to be processed into a plurality of subsets includes:

[0021] Standardizing the data to be processed to obtain row and column data of a two-dimensional table structure;

[0022] A preset space size is obtained, and each row in the row and column data is used as a unit, and the row and column data is divided based on the preset space size to obtain a plurality of subsets.

[0023] Optionally, determining the storage node and the replication node corresponding to each subset from the computing nodes includes:

[0024] Obtaining the number of computing nodes;

[0025] The plurality of subsets are evenly distributed to the computing nodes based on the number of the computing nodes, and at least one storage node corresponding to each subset is determined.

[0026] Optionally, after dividing the data to be processed into a plurality of subsets, the method further includes:

[0027] Performing feature analysis on the subset and constructing metadata;

[0028] Constructing index information according to the data type and distribution of the subset;

[0029] The allocating the subsets to the corresponding storage nodes respectively includes:

[0030] The subset, the metadata, and the index information are allocated to the corresponding storage nodes.

[0031] Optionally, before aggregating the received subtask processing results to obtain the target processing result, the method further includes:

[0032] If the subtask processing result of any subtask is not received within a preset time, determining the fault-tolerant node corresponding to any subtask from the computing nodes;

[0033] The subtask is assigned to a corresponding fault-tolerant node, so that the fault-tolerant node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server.

[0034] In a second aspect, an embodiment of the present application provides a distributed data processing method, which is applied to a storage node in a distributed system, and the method includes:

[0035] Dividing the memory of the storage node into a plurality of memory pages;

[0036] The subset, metadata and index information assigned by the server are received and stored, and the index information and the metadata are shared in the distributed system.

[0037] Optionally, the receiving and storing the subset, metadata, and index information assigned by the server includes:

[0038] Receive the subset, metadata and index information assigned by the server;

[0039] Performing compression encoding on the subset to obtain encoded data;

[0040] The encoded data, the metadata, and the index information are stored in the memory page.

[0041] In a third aspect, an embodiment of the present application provides a distributed data processing method, which is applied to a task node in a distributed system, and the method includes:

[0042] Receive subtasks assigned by the server;

[0043] Determine a storage node storing task data in the distributed system, and obtain the task data from the storage node; the task data is the data to be processed corresponding to the subtask;

[0044] According to the subtask, the task data is processed to obtain a subtask processing result, and the subtask processing result is returned to the server.

[0045] Optionally, there are multiple storage nodes storing the task data; and obtaining the task data from the storage nodes includes:

[0046] Determine a target node from the storage nodes, and obtain the task data from the target node;

[0047] In the case that the task data fails to be obtained from the target node, the task data is obtained from other storage nodes except the target node.

[0048] In a fourth aspect, an embodiment of the present application provides a distributed data processing device, which is applied to a server in a distributed system, wherein the distributed system also includes a plurality of computing nodes, including:

[0049] An acquisition module, used for acquiring a data processing task for data to be processed; the data to be processed is distributedly stored in a plurality of computing nodes;

[0050] A task splitting module, used to divide the data processing task into multiple subtasks, and determine the task node corresponding to each subtask from the computing nodes;

[0051] A task assignment module, used to assign the subtask to a corresponding task node, so that the task node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server;

[0052] The aggregation module is used to aggregate the received subtask processing results to obtain the target processing results.

[0053] In a fifth aspect, an embodiment of the present application provides a distributed data processing device, which is applied to a storage node in a distributed system, and the device includes:

[0054] A memory segmentation module, used to segment the memory of the storage node into multiple memory pages;

[0055] A storage module, used for receiving and storing the subset, metadata and index information assigned by the server;

[0056] A sharing module is used to share the index information and the metadata in the distributed system.

[0057] In a sixth aspect, an embodiment of the present application provides a distributed data processing device, which is applied to a task node in a distributed system, and the device includes:

[0058] A receiving module, used for receiving subtasks assigned by the server;

[0059] An acquisition module, used to determine a storage node storing task data in the distributed system, and acquire the task data from the storage node; the task data is the data to be processed corresponding to the subtask;

[0060] The processing module is used to process the task data according to the subtask, obtain the subtask processing result, and return the subtask processing result to the server.

[0061] In a seventh aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0062] In an eighth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0063] In a ninth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0064] In a tenth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0065] In the solution provided in the embodiment of the present application, the distributed system includes a server and multiple computing nodes, wherein the server can obtain a data processing task for the data to be processed; the data to be processed is distributed and stored in multiple computing nodes; the data processing task is divided into multiple subtasks, and the task node corresponding to each subtask is determined from the computing node; the subtask is assigned to the corresponding task node, so that the task node obtains the data to be processed corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the data to be processed corresponding to the subtask to the server; the received subtask processing results are aggregated to obtain the target processing result.

[0066] As can be seen from the above, by adopting a distributed architecture, multiple computing nodes can be connected in a distributed system. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flow chart of a distributed data processing method according to an exemplary embodiment;

[0068] Figure 2 is a flow chart showing another distributed data processing method according to an exemplary embodiment;

[0069] Figure 3 is a flow chart showing another distributed data processing method according to an exemplary embodiment;

[0070] Figure 4 is a diagram showing an architecture of a distributed data processing system according to an exemplary embodiment;

[0071] Figure 5 is a schematic diagram of data segmentation according to an exemplary embodiment;

[0072] Figure 6 is a schematic diagram of a network scenario according to an exemplary embodiment;

[0073] Figure 7 is a data processing flow chart according to an exemplary embodiment;

[0074] Figure 8 is a block diagram of a distributed data processing device according to an exemplary embodiment;

[0075] Fig. 9 is a block diagram of another distributed data processing device according to an exemplary embodiment;

[0076] Fig.10 is a block diagram of another distributed data processing device according to an exemplary embodiment;

[0077] Fig.11 A block diagram of an electronic device according to an exemplary embodiment;

[0078] Fig.12 The figure is a schematic diagram showing the hardware structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0080] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0081] The distributed data processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0082] Figure 1 It is a flowchart of a distributed data processing method according to an exemplary embodiment, which is applied to a server in a distributed system. The distributed system also includes multiple computing nodes. The distributed data processing method includes the following steps.

[0083] In step S11, a data processing task for the data to be processed is obtained; the data to be processed is distributedly stored in multiple computing nodes.

[0084] Currently, the data processing software in the relevant technology has certain restrictions on the number of rows of data that can be loaded. It cannot load data sets exceeding the preset number of rows. Therefore, it cannot support large-scale data visualization analysis and cannot process large-scale data sets. Moreover, data processing software is usually run on a single machine. Due to the processing power of a single machine, when processing data sets with a slightly larger data scale, it is easy to have abnormal situations such as operation jams and slow responses. These problems have seriously hindered the application of spreadsheet processing software.

[0085] Based on this, the present application proposes a distributed data processing method to solve the above problem. The present application can be applied to a server in a distributed system, that is, a master computing node, and the distributed system also includes multiple computing nodes, that is, slave computing nodes corresponding to the master computing node.

[0086] First, in this step, the server obtains data processing tasks for the data to be processed, wherein the data processing tasks include complex operation tasks such as data aggregation, grouping, and sorting of the data to be processed, and are not specifically limited.

[0087] In this application, the data to be processed is distributed and stored in multiple computing nodes. Therefore, by taking advantage of the distributed system, the data to be processed can be a large-scale and computationally complex data set, such as data sets in the fields of big data analysis and data mining. Alternatively, it can also be data sets in multiple other industries and fields, such as information technology, communication technology, engineering technology, etc., to meet the needs of different fields.

[0088] In one implementation, before obtaining a data processing task for the data to be processed, the process includes:

[0089] Get the data to be processed;

[0090] Divide the data to be processed into multiple subsets, determine at least one storage node corresponding to each subset from the computing nodes, and allocate the subsets to the corresponding storage nodes respectively;

[0091] The task node obtains the to-be-processed data corresponding to the subtask from any storage node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server.

[0092] That is, a distributed memory computing framework can be used to store the data to be processed to achieve fast data access and computing. In order to efficiently process large amounts of data to be processed in a distributed system, the data to be processed must first be divided into multiple subsets and distributed to different storage nodes.

[0093] A distributed memory management mechanism can be used in a distributed system. Each storage node maintains its own local memory and realizes sharing and access of data stored in the local memory through network communication. This can effectively utilize the memory resources of the entire system and avoid the problem of insufficient memory of a single node.

[0094] Among them, the storage node corresponding to each subset can be determined randomly, or the computing node with less memory occupancy or less network load can be selected as the storage node based on the current memory occupancy or network load of each computing node. This application does not limit this.

[0095] The number of storage nodes corresponding to each subset can be one or more. In this way, redundant storage can be provided for each subset. When an error occurs in the subset stored in one of the storage nodes or the storage node fails, the data of the subset can be obtained from another storage node, thereby improving the fault tolerance of the distributed system.

[0096] In one implementation, the data to be processed is divided into multiple subsets, including:

[0097] Standardize the data to be processed to obtain row and column data of a two-dimensional table structure;

[0098] A preset space size is obtained, and each row in the row and column data is used as a unit, and the row and column data is divided based on the preset space size to obtain multiple subsets.

[0099] That is to say, first the data to be processed is normalized by columns and processed into row and column data of a standard two-dimensional table structure. Then, in order to improve the concurrent processing capability, the row and column data is split by rows, and several rows are divided into a subset, and the size of each subset does not exceed the preset space size. In this way, multiple subsets are obtained.

[0100] In one implementation, determining the storage node and the replication node corresponding to each subset from the computing nodes includes:

[0101] Get the number of computing nodes;

[0102] Based on the number of computing nodes, multiple subsets are evenly distributed to the computing nodes, and at least one storage node corresponding to each subset is determined.

[0103] In other words, the subsets can be evenly distributed to each computing node. If the number of subsets is greater than the number of computing nodes, then the storage resources of each computing node can be more reasonably applied through average distribution, avoiding the failure of the distributed system caused by excessive memory usage of a computing node.

[0104] In the present application, a distribution table may be maintained in the server, and the distribution status of the data to be processed may be recorded in the distribution table.

[0105] For example, as shown in Table 1, it is an allocation table for evenly dividing the data to be processed and allocating them to three computing nodes.

[0106] Table 1 Allocation table

[0107] Subset number Row number of row and column data First storage node number Second storage node number 1 1:n 1 2 2 n+1:2n 2 3 3 2n+1:3n 3 1 4 3n+1:4n 1 2 5 4n+1:5n 2 3 m n+1:mn 3 1

[0108] The data to be processed is divided into m subsets, the first subset includes the first row to the nth row of the row and column data, the second subset includes the n+1th row to the 2nth row of the row and column data, the third subset includes the 2n+1th row to the 3nth row of the row and column data, and so on. Moreover, each subset is stored in two different storage nodes, for example, the first subset is stored in two storage nodes numbered 1 and 2, the second subset is stored in two storage nodes numbered 2 and 3, and so on.

[0109] In one implementation, after dividing the data to be processed into a plurality of subsets, the method further includes:

[0110] Perform feature analysis on the subset and construct metadata;

[0111] Construct index information based on the data type and distribution of the subset;

[0112] Allocate the subsets to the corresponding storage nodes, including:

[0113] Allocate subsets, metadata, and index information to corresponding storage nodes.

[0114] That is to say, while dividing the data to be processed, the characteristics of each subset can be preprocessed and analyzed to construct the metadata information of the subset, including the data type information of each column (numeric value, text), the number of data rows, the distribution of discrete data, etc. In addition, the index data of the subset can be constructed so that the storage node can adopt different data storage compression methods according to the different data types of the subset. By compressing and storing each subset by the storage node, the memory usage and data transmission overhead can be reduced, thereby improving the memory utilization and computing speed.

[0115] In step S12, the data processing task is divided into a plurality of subtasks, and a task node corresponding to each subtask is determined from the computing nodes.

[0116] In this step, multithreading and parallel computing technologies can be used to decompose the data processing task into multiple subtasks, and these subtasks are assigned to different task nodes for simultaneous execution to improve computing efficiency.

[0117] Moreover, node failure is a common occurrence in distributed systems. Therefore, a mechanism is needed to ensure that even if some task nodes fail, the data processing task can still be completed and the results remain consistent. Therefore, data replication, checkpoint mechanism and fault recovery strategy can be used to handle the allocation of subtasks, which is not limited in this application.

[0118] In step S13, the subtask is assigned to the corresponding task node, so that the task node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server.

[0119] In this step, after the subtask is assigned to the corresponding task node, the corresponding task node can obtain the data to be processed corresponding to the subtask from the computing node that stores the data to be processed, and process the data to be processed corresponding to the subtask to obtain the subtask processing result. Then, the task node can return the subtask processing result to the server.

[0120] Specifically, the MapReduce (map-reduce) model can be used to implement the allocation and aggregation of data processing tasks. The MapReduce model includes the Map function and the Reduce function. The Map function is used to coordinate each task node to perform parallel task processing on the corresponding data to be processed according to the assigned subtasks. The Reduce function is used to aggregate the output results of the Map function.

[0121] In this application, the server can set a distributed lock to ensure data integrity during the computing process. Distributed locks are a way to control the synchronous access to shared resources between distributed systems. It can be understood that in a distributed system, various computing nodes may share one or a group of resources. In order to prevent these resources from interfering with each other when sharing and ensure data consistency, distributed locks are needed to coordinate their actions to ensure that only one process or thread can access shared resources at any time, thereby avoiding data conflicts and inconsistencies.

[0122] After each task node completes the calculation, it automatically reports the subtask processing results to the server until all computing nodes have completed the calculation. Each node reports to the master node, and the master node performs global aggregation and calculation on the data.

[0123] Among them, data exchange and coordinated control between computing nodes will generate communication overhead, which may become a performance bottleneck. Reducing unnecessary data transmission, adopting efficient message passing protocols, and optimizing network topology are all effective means to reduce communication costs.

[0124] Specifically, communication between computing nodes is the key to achieving distributed computing. Efficient communication protocols and data transmission mechanisms are used to ensure the rapid transmission and synchronization of data between nodes. At the same time, a collaborative mechanism is used to ensure that each node can work together to complete the perspective computing task. In the distributed computing process, it is crucial to ensure data consistency. Data synchronization mechanisms, distributed locks, data replication or consistent hashing can be used to ensure that each computing node can maintain data consistency when processing data. The calculation process is optimized for different data processing tasks, and distributed aggregation algorithms are used to accelerate data aggregation and calculation. Compression storage and indexing technologies are used to reduce data transmission and storage overhead, and cache mechanisms are used to reduce repeated calculations.

[0125] In step S14, the received subtask processing results are aggregated to obtain the target processing result.

[0126] That is to say, each task node sends the subtask processing results to the server for merging, and the server is responsible for summarizing and integrating the subtask processing results of each task node to generate the final target processing results.

[0127] For example, for tasks such as data grouping, aggregation, and averaging, the server can aggregate all result sets. For tasks such as sorting, the server only stores the index data for record sorting, and the original row data is stored in the computing nodes. When full export is required, the data is directly exported from each computing node through the index.

[0128] In one implementation, obtaining a data processing task for data to be processed includes:

[0129] Get the data processing request sent by the user;

[0130] Parse the data processing request to obtain the data processing task for the data to be processed;

[0131] After aggregating the received subtask processing results to obtain the target processing results, it also includes:

[0132] The target processing result is returned to the user end for display.

[0133] That is to say, the user can interact with the user terminal and input data processing instructions, and then the user terminal sends the corresponding data processing request. Then, after obtaining the target processing result, the server can return the target processing result to the user terminal to realize the visualization of data processing.

[0134] In this application, although the background is complex distributed computing, the front-end interface on the user side needs to provide users with an intuitive and easy-to-use experience, as simple as processing data locally. This includes quickly responding to users' changes in pivot table settings, and providing clear error reports and usage instructions.

[0135] For example, setting up the user interface may include:

[0136] Title bar: Displays the product name or logo, and the title of the current page or function, such as "Online Spreadsheet Perspective".

[0137] Menu bar: provides a series of functional options, such as "File", "Edit", "Perspective", etc. Users can perform different operations by clicking menu items.

[0138] Toolbar: Displays a series of commonly used tool buttons, such as "New", "Save", "Refresh", etc. These buttons can quickly trigger some common operations and improve user work efficiency.

[0139] Spreadsheet area: Displays the cells and data of the spreadsheet. Users can select data ranges, set pivot fields, etc. in this area.

[0140] Pivot Options Panel: Provides setting options for pivot operations, such as row labels, column labels, value fields, etc. Users can customize the pivot results by selecting different options.

[0141] Result display area: Displays the results of the pivot operation. The results can be grouped, aggregated, and displayed according to the pivot options set by the user.

[0142] Status bar: Displays the status information of the current operation, such as data processing progress, error prompts, etc. Helps users understand the completion status of the current operation or possible problems.

[0143] In one implementation, before aggregating the received subtask processing results to obtain the target processing result, the method further includes:

[0144] If the subtask processing result of any subtask is not received within a preset time, a fault-tolerant node corresponding to any subtask is determined from the computing nodes;

[0145] The subtasks are assigned to the corresponding fault-tolerant nodes, so that the fault-tolerant nodes obtain the to-be-processed data corresponding to the subtasks from the computing nodes, and return the subtask processing results obtained by processing the to-be-processed data corresponding to the subtasks to the server.

[0146] That is to say, the task node can set a fault-tolerant mechanism for the processing of the subtask. After the server does not receive the subtask processing result within a preset time, the spare task node will be enabled to process the subtask.

[0147] In this application, the distributed system needs to have powerful monitoring and diagnostic tools to track the system status in real time, quickly identify performance bottlenecks, and adjust the computing strategy based on the feedback.

[0148] That is to say, in this application, distributed memory computing technology is used to process and analyze large-scale data to be processed, avoiding frequent hard disk read and write operations and greatly improving the computing speed; using an in-memory database or a distributed memory computing framework to store and process data to achieve fast data access and calculation; designing an efficient perspective computing algorithm that can quickly aggregate and summarize data in memory, reducing unnecessary computing steps and data movement; using multi-threading and parallel computing technology, some computing tasks are decomposed into multiple subtasks and executed simultaneously to improve computing efficiency; compressing and storing data to reduce memory usage and data transmission overhead, thereby improving memory utilization and computing speed; carefully designing memory management strategies to avoid memory leaks and unnecessary memory usage, and ensure stable and efficient system operation.

[0149] In addition, the results of some frequently queried queries are pre-calculated and cached. When the user requests the same data, the cached target processing results can be directly returned to reduce the calculation time; the intermediate calculation results are stored using caching technology to avoid repeated calculations and data reading, thereby improving data processing efficiency.

[0150] For real-time data updates of data to be processed, corresponding data processing and calculation mechanisms can be designed to generate perspective data in real time and feed it back to users when the data is updated; combined with streaming processing technology, rapid processing and perspective calculation of dynamic data can be achieved, so that users can obtain the latest statistical results at any time.

[0151] Through data processing algorithms, memory management technology, parallel computing, etc., a method of accelerated data processing based on memory computing is implemented, which can achieve second-level operation effects on a scale of tens of millions, and improve the existing data processing efficiency by more than 80%.

[0152] As can be seen from the above, by adopting a distributed architecture, multiple computing nodes can be connected in a distributed system. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0153] Figure 2 The present invention is a flow chart of a distributed data processing method according to an exemplary embodiment, which is applied to a storage node in a distributed system. The distributed data processing method includes the following steps:

[0154] S21: Divide the memory of the storage node into multiple memory pages.

[0155] In this step, it is crucial to effectively manage the memory resources of the storage nodes, which includes reasonably allocating the memory space of each storage node, avoiding memory overflow of some nodes caused by data skew, and designing efficient caching strategies to speed up data access.

[0156] That is to say, each storage node first pre-allocates the local memory block, divides the memory into pages of fixed size, forms memory page information, and each storage node manages its own memory page usage information.

[0157] S22: Receive and store the subset, metadata, and index information assigned by the server, and share the index information and metadata in the distributed system.

[0158] That is, after receiving the subset, metadata, and index information assigned by the server, the storage node can start the process of loading the data file into the memory. The storage node loads the received subset into the memory block and reports the loading completion status to the allocation table. Each storage node can share the necessary metadata and index information through network communication.

[0159] In one implementation, receiving and storing the subset, metadata, and index information assigned by the server includes:

[0160] Receive the subset, metadata and index information assigned by the server;

[0161] Perform compression encoding on the subset to obtain encoded data;

[0162] Stores the encoded data, metadata, and index information into memory pages.

[0163] That is to say, data can be compressed when loaded into the memory of the storage node to reduce memory overhead. For example, the dictionary encoding compression algorithm is used for text type column information, and repeated text information is replaced with a variable-length integer sequence.

[0164] In this application, distributed memory computing technology is used to process and analyze large-scale data to be processed, avoiding frequent hard disk read and write operations and greatly improving the computing speed; using an in-memory database or a distributed memory computing framework to store and process data to achieve fast data access and calculation; designing an efficient perspective computing algorithm that can quickly aggregate and summarize data in memory, reducing unnecessary computing steps and data movement; compressing and storing data to reduce memory usage and data transmission overhead, thereby improving memory utilization and computing speed; carefully designing memory management strategies to avoid memory leaks and unnecessary memory usage, ensuring stable and efficient system operation.

[0165] As can be seen from the above, by adopting a distributed architecture, multiple computing nodes can be connected in a distributed system. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0166] Figure 3 The present invention is a flow chart of a distributed data processing method according to an exemplary embodiment, which is applied to a task node in a distributed system. The distributed data processing method includes the following steps:

[0167] S31: receiving the subtask assigned by the server.

[0168] That is to say, multi-threading and parallel computing technologies can be used to decompose some computing tasks into multiple sub-tasks and execute them simultaneously, wherein a task node can receive a sub-task assigned by a server and execute the sub-task to improve computing efficiency.

[0169] S32: Determine a storage node storing task data in the distributed system, and obtain the task data from the storage node; the task data is the data to be processed corresponding to the subtask.

[0170] In a distributed system, each computing node can access each other. Therefore, after receiving a subtask, the task node can determine the storage node that stores the task data corresponding to the subtask, and then obtain the task data from the corresponding storage node.

[0171] In one implementation, the storage nodes storing the task data include a plurality of nodes; obtaining the task data from the storage nodes includes:

[0172] Determine the target node from the storage node and obtain the task data from the target node;

[0173] In the case that the task data fails to be obtained from the target node, the task data is obtained from other storage nodes except the target node.

[0174] That is to say, redundant storage can be performed when data is stored. In this way, if one of the storage nodes storing task data fails, or the task data stored in a storage node is damaged, the task data can be obtained from another storage node, thereby improving the fault tolerance of the distributed system.

[0175] S33: Process the task data according to the subtask, obtain the subtask processing result, and return the subtask processing result to the server.

[0176] After the task node obtains the subtask processing result of the subtask, it can return the subtask processing result to the server. The server will aggregate the subtask processing results returned by each task node to obtain the final target processing result.

[0177] As can be seen from the above, the technical solution provided by the embodiment of the present application can connect multiple computing nodes in a distributed system by adopting a distributed architecture. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0178] The data processing method provided by the present application is described below by taking online electronic spreadsheet processing software for processing table data as an example.

[0179] like Figure 4 As shown, it is a distributed data processing system architecture diagram provided by an embodiment of the present application, including:

[0180] Interface operation: responsible for interface processing control, interacting with users, and receiving processing tasks for online spreadsheet perspectives;

[0181] Server-side coordination management: used to divide the processing tasks of online spreadsheet pivoting and split the data set that needs pivoting into multiple subsets. Each subset is assigned to one or more computing nodes for processing. This can fully utilize the parallel processing capabilities of the distributed computing environment and improve the computing speed.

[0182] Computing node cluster: includes multiple computing nodes, adopts distributed memory management mechanism, each node maintains its own local memory and cache, and realizes sharing and access of memory data through network communication, which can effectively utilize the memory resources of the whole system and avoid the problem of insufficient memory of a single node; communication between nodes is the key to realize distributed computing, adopts efficient communication protocol and data transmission mechanism to ensure fast transmission and synchronization of data between nodes, and at the same time, through the coordination mechanism, ensures that each node can work together to complete the perspective computing task.

[0183] like Figure 5 As shown, it is a data segmentation schematic diagram in an embodiment of the present application. In order to efficiently process large electronic spreadsheet data in a distributed system, it is first necessary to segment the data to be processed into multiple subsets and distribute them to different computing nodes, with each file corresponding to a processing process. For example, the data file of the data to be processed can be segmented into three parts: file block 1, file block 2, and file block 3, wherein file block 1, file block 2, and file block 3 correspond to processing process 1, processing process 2, and processing process 3, respectively, and then respectively distributed to processor 1, processor 2, and processor 3. This requires the design of a reasonable data segmentation strategy to ensure that the data is evenly divided and the correlation between the data can be minimized to facilitate parallel processing.

[0184] like Figure 6 FIG. 1 is a schematic diagram of a network scenario in an embodiment of the present application, which includes:

[0185] 1. User side:

[0186] Users access the online spreadsheet pivot system through a browser or a dedicated client.

[0187] The client sends data query, perspective request and other instructions to the application server.

[0188] 2. Application Server:

[0189] Receive requests from the user and parse them into specific computing tasks.

[0190] Communicate with computing nodes to coordinate and distribute computing tasks.

[0191] Receive the calculation results of the computing nodes, perform necessary processing and return them to the user end.

[0192] 3. Computing node cluster:

[0193] It contains multiple computing nodes, each of which has independent computing and storage capabilities.

[0194] Computing nodes communicate through high-speed networks to achieve data sharing and task collaboration.

[0195] Each computing node performs specific spreadsheet perspective calculation tasks according to the assignment of the task scheduler.

[0196] 4. Distributed storage system:

[0197] Responsible for storing original data, intermediate results, and final perspective results.

[0198] Provides efficient data access and transmission mechanisms to ensure that computing nodes can quickly obtain required data.

[0199] It can communicate with other components through the network to achieve data backup, recovery and fault-tolerant processing.

[0200] 5. Network facilities:

[0201] Including network devices such as routers, switches, firewalls, etc., to ensure unimpeded network connections between various components.

[0202] Provide high-speed, low-latency network transmission services, supporting the rapid transmission and processing of large amounts of data.

[0203] like Figure 7 FIG. 1 is a data processing flow chart in an embodiment of the present application, including:

[0204] 1. User Request:

[0205] Users initiate pivot requests through the online spreadsheet interface, including selecting pivot fields, filtering conditions, etc.

[0206] 2. Request processing:

[0207] After receiving the user request, the application server parses and processes it and generates a specific computing task.

[0208] The application server assigns computing tasks to idle nodes in the computing node cluster.

[0209] 3. Data reading:

[0210] The computing nodes read the required raw data from the distributed storage system.

[0211] According to the requirements of the perspective request, the computing node performs pre-processing operations such as filtering and aggregation on the data.

[0212] 4. Distributed computing:

[0213] The computing nodes use distributed memory computing technology to perform parallel computing on the preprocessed data.

[0214] Accelerate data access and communication between computing nodes through memory sharing or memory replication mechanisms.

[0215] 5. Summary of results:

[0216] The computing nodes aggregate their respective computing results to the master node or the designated aggregation node.

[0217] The Aggregate node performs the final merging and sorting of the results to produce the final spreadsheet pivot results.

[0218] 6. Results returned:

[0219] The aggregation node returns the pivoted results to the application server.

[0220] The application server performs necessary formatting on the results and displays them to the user through an online spreadsheet interface.

[0221] As can be seen from the above, by adopting a distributed architecture, multiple computing nodes can be connected in a distributed system. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0222] The distributed data processing method provided in the embodiment of the present application can be executed by a distributed data processing device. In the embodiment of the present application, a method in which a distributed data processing device executes terminal access is taken as an example to illustrate the device of the distributed data processing method provided in the embodiment of the present application.

[0223] Figure 8 The present invention is a block diagram of a distributed data processing device according to an exemplary embodiment, which is applied to a server in a distributed system. The distributed system also includes multiple computing nodes, including:

[0224] An acquisition module 401 is used to acquire a data processing task for data to be processed; the data to be processed is distributedly stored in a plurality of computing nodes;

[0225] A task division module 402 is used to divide the data processing task into multiple subtasks and determine the task node corresponding to each subtask from the computing nodes;

[0226] The task assignment module 403 is used to assign the subtask to the corresponding task node, so that the task node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server;

[0227] Aggregation module 404 is used to aggregate the received subtask processing results to obtain target processing results.

[0228] As can be seen from the above, by adopting a distributed architecture, multiple computing nodes can be connected in a distributed system. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0229] Fig. 9 The present invention is a block diagram of a distributed data processing device according to an exemplary embodiment, which is applied to a storage node in a distributed system. The device includes:

[0230] A memory segmentation module 501, used to segment the memory of the storage node into multiple memory pages;

[0231] Storage module 502, used to receive and store subsets, metadata and index information assigned by the server;

[0232] The sharing module 503 is used to share the index information and the metadata in the distributed system.

[0233] As can be seen from the above, the technical solution provided by the embodiment of the present application can connect multiple computing nodes in a distributed system by adopting a distributed architecture. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0234] Fig.10The present invention is a block diagram of a distributed data processing device according to an exemplary embodiment, which is applied to a task node in a distributed system. The device includes:

[0235] Receiving module 601, used for receiving the subtask assigned by the server;

[0236] An acquisition module 602 is used to determine a storage node storing task data in the distributed system, and acquire the task data from the storage node; the task data is the data to be processed corresponding to the subtask;

[0237] The processing module 603 is used to process the task data according to the subtask, obtain the subtask processing result, and return the subtask processing result to the server.

[0238] As can be seen from the above, by adopting a distributed architecture, multiple computing nodes can be connected in a distributed system. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0239] The distributed data processing method provided in the embodiment of the present application can be executed by a terminal access terminal. The method of executing terminal access by a terminal access terminal is taken as an example to illustrate the apparatus of the distributed data processing method provided in the embodiment of the present application.

[0240] The distributed data processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0241] The distributed data processing device provided in the embodiment of the present application can realize Figures 1 to 7 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0242] Alternatively, if Fig.11 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the above-mentioned distributed data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0243] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0244] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0245] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0246] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0247] As can be seen from the above, the technical solution provided by the embodiment of the present application can connect multiple computing nodes in a distributed system by adopting a distributed architecture. On the one hand, the data to be processed can be distributed and stored in each computing node. On the other hand, the data processing task can be divided into multiple subtasks, and each task node executes each subtask synchronously, thereby realizing the distributed storage and calculation of the data to be processed. Therefore, the data processing software based on the distributed system of the present application can process large-scale data to be processed, and can make full use of the computing power of each computing node, and is not prone to abnormal situations such as operation jams and slow responses, thereby having higher computing efficiency.

[0248] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0249] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0250] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0251] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned distributed data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0252] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0253] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned distributed data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0254] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0255] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned distributed data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0256] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0257] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0258] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A distributed data processing method, characterized in that: The method is applied to a server in a distributed system, wherein the distributed system further comprises a plurality of computing nodes, and comprises: Acquire a data processing task for the data to be processed; the data to be processed is distributed and stored in a plurality of the computing nodes; Divide the data processing task into a plurality of subtasks, and determine a task node corresponding to each subtask from the computing nodes; Allocating the subtask to a corresponding task node so that the task node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server; The received subtask processing results are aggregated to obtain the target processing result.

2. The distributed data processing method according to claim 1, characterized in that: The data processing task of obtaining the data to be processed includes: Get the data processing request sent by the user; Parsing the data processing request to obtain a data processing task for the data to be processed; After aggregating the received subtask processing results to obtain the target processing results, the method further includes: The target processing result is returned to the user terminal for display.

3. The distributed data processing method according to claim 1, characterized in that: Before obtaining the data processing task for the data to be processed, the following steps are included: Get the data to be processed; Divide the data to be processed into multiple subsets, determine at least one storage node corresponding to each subset from the computing nodes, and allocate the subsets to the corresponding storage nodes respectively; The task node obtains the to-be-processed data corresponding to the subtask from any of the storage nodes, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server.

4. The distributed data processing method according to claim 3, characterized in that: The step of dividing the data to be processed into a plurality of subsets comprises: Standardizing the data to be processed to obtain row and column data of a two-dimensional table structure; A preset space size is obtained, and each row in the row and column data is used as a unit, and the row and column data is divided based on the preset space size to obtain a plurality of subsets.

5. The distributed data processing method according to claim 3, characterized in that: The step of determining the storage node and the replication node corresponding to each subset from the computing nodes includes: Obtaining the number of computing nodes; The plurality of subsets are evenly distributed to the computing nodes based on the number of the computing nodes, and at least one storage node corresponding to each subset is determined.

6. The distributed data processing method according to claim 3, characterized in that: After dividing the data to be processed into a plurality of subsets, the method further includes: Performing feature analysis on the subset and constructing metadata; Constructing index information according to the data type and distribution of the subset; The allocating the subsets to the corresponding storage nodes respectively includes: The subset, the metadata, and the index information are allocated to the corresponding storage nodes.

7. The distributed data processing method according to claim 1, characterized in that: Before aggregating the received subtask processing results to obtain the target processing results, the method further includes: If the subtask processing result of any subtask is not received within a preset time, determining the fault-tolerant node corresponding to any subtask from the computing nodes; The subtask is assigned to a corresponding fault-tolerant node, so that the fault-tolerant node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server.

8. A distributed data processing method, characterized in that: Applied to a storage node in a distributed system, the method comprises: Dividing the memory of the storage node into a plurality of memory pages; The subset, metadata and index information assigned by the server are received and stored, and the index information and the metadata are shared in the distributed system.

9. The distributed data processing method according to claim 8, characterized in that: The receiving and storing of the subset, metadata and index information assigned by the server includes: Receive the subset, metadata and index information assigned by the server; Performing compression encoding on the subset to obtain encoded data; The encoded data, the metadata, and the index information are stored in the memory page.

10. A distributed data processing method, characterized in that: Applied to a task node in a distributed system, the method comprises: Receive subtasks assigned by the server; Determine a storage node storing task data in the distributed system, and obtain the task data from the storage node; the task data is the data to be processed corresponding to the subtask; According to the subtask, the task data is processed to obtain a subtask processing result, and the subtask processing result is returned to the server.

11. The distributed data processing method according to claim 10, characterized in that: The storage nodes storing the task data include a plurality of nodes; and obtaining the task data from the storage nodes includes: Determine a target node from the storage nodes, and obtain the task data from the target node; In the case that the task data fails to be obtained from the target node, the task data is obtained from other storage nodes except the target node.

12. A distributed data processing device, characterized in that: A server applied to a distributed system, wherein the distributed system further includes a plurality of computing nodes, including: An acquisition module, used for acquiring a data processing task for data to be processed; the data to be processed is distributedly stored in a plurality of computing nodes; A task splitting module, used to divide the data processing task into multiple subtasks, and determine the task node corresponding to each subtask from the computing nodes; A task assignment module, used to assign the subtask to a corresponding task node, so that the task node obtains the to-be-processed data corresponding to the subtask from the computing node, and returns the subtask processing result obtained by processing the to-be-processed data corresponding to the subtask to the server; The aggregation module is used to aggregate the received subtask processing results to obtain the target processing results.

13. A distributed data processing device, characterized in that: Applied to a storage node in a distributed system, the device comprises: A memory segmentation module, used to segment the memory of the storage node into multiple memory pages; A storage module, used for receiving and storing the subset, metadata and index information assigned by the server; A sharing module is used to share the index information and the metadata in the distributed system.

14. A distributed data processing device, characterized in that: Applied to a task node in a distributed system, the device comprises: A receiving module, used for receiving subtasks assigned by the server; An acquisition module, used to determine a storage node storing task data in the distributed system, and acquire the task data from the storage node; the task data is the data to be processed corresponding to the subtask; The processing module is used to process the task data according to the subtask, obtain the subtask processing result, and return the subtask processing result to the server.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the distributed data processing method according to any one of claims 1 to 11.

16. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the distributed data processing method according to any one of claims 1 to 11 are implemented.

Images