Big data engine system, data processing method based on Shuffle process and electronic equipment
By using preset data merging algorithm in the big data engine system to merge data into fewer storage shards and store metadata, the problem of low data processing efficiency in the Shuffle stage is solved, and the analysis efficiency and resources of the big data computing engine are improved. Utilization.
Patent Information
- Application Number
- CN202410989588.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-12-03
AI Technical Summary
The existing MapReduce data processing technology has the problem of low data processing efficiency in the Shuffle stage, which leads to frequent disk IO and network resource consumption, affecting the data analysis efficiency of the big data computing engine.
A big data engine system and a data processing method based on Shuffle process are designed. The data calculated by Map are merged into the same target storage node according to the key value range through a preset data merging algorithm, and stored in the Shuffle metadata management service. Index information to reduce the number of data shards, reduce disk access frequency and network resource consumption.
It effectively improves the data analysis efficiency of the big data computing engine, reduces the IO and network resource usage in the Shuffle stage, and improves the speed and accuracy of data processing.
Smart Images

Figure CN119066032A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of big data processing, and in particular to a big data engine system, a data processing method based on the Shuffle process, and an electronic device. Background Art
[0002] In the field of big data processing technology, especially in the scenario of distributed computing, big data computing engines (such as Spark big data computing engine, Hive big data computing engine, Hadoop big data computing engine) are usually used to perform some offline data analysis tasks. When the big data computing engine performs offline data analysis tasks, it usually uses the MapReduce data processing process to read the required data, or uses the MapReuduce data processing process to write data to the backend storage device.
[0003] The existing MapReduce data processing process usually includes: Map phase (mapping phase), Shuffle phase (shuffle phase), Reduce phase (reduction phase). However, in the existing MapReduce data processing technology, the data passed in from the Map phase is often written into each Partition data shard according to the Partition key-value mapping relationship in the Shuffle phase, and each data shard is located in a different storage node. When the data needs to be read later, the corresponding data needs to be obtained from each Partition data shard.
[0004] In this way, the entire Shuffle stage requires frequent data reading and writing across storage nodes, which easily occupies a large amount of disk IO (Input / Output), consumes a large amount of network resources and system resources, and further affects the data analysis efficiency of the big data computing engine. Summary of the invention
[0005] In view of this, the embodiments of the present application provide a big data engine system, a data processing method based on the Shuffle process, and an electronic device to solve the problem of low data processing efficiency in the existing Shuffle stage.
[0006] In a first aspect, an embodiment of the present application provides a big data engine system, wherein the big data engine system at least includes:
[0007] The client is used to receive offline data analysis tasks, wherein the offline data analysis tasks include: Shuffle data reading tasks and Shuffle data writing tasks;
[0008] The Shuffle processing service module is used to merge and process each of the first pre-processed data based on the key value of each of the first pre-processed data using a preset data merging algorithm, wherein the first pre-processed data is: the data located on different data shards obtained after the offline data analysis task Map calculation is completed; wherein the preset data merging algorithm is used to merge and map the first pre-processed data located in the same key value range to the same target storage node based on the key value of each of the first pre-processed data;
[0009] A Shuffle data storage service module, configured to store each of the first pre-processed data in a corresponding target storage node according to a key value of the target storage node to which each of the first pre-processed data belongs;
[0010] The Shuffle metadata management service module is used to store metadata of each of the first pre-processed data, where the metadata is storage index information of each of the first pre-processed data in each of the target storage nodes.
[0011] Optionally, the Shuffle data reading task carries request information of target first pre-processed data to be read;
[0012] The Shuffle metadata management service module is further used to transmit the metadata of the target first pre-processed data to the Shuffle data storage service module in response to the target first pre-processed data request information;
[0013] The Shuffle data storage service module is used to obtain the target first preprocessed data based on the metadata of the target first preprocessed data, and return the target first preprocessed data to the client.
[0014] Optionally, the system further includes:
[0015] The Shuffle task scheduling module is used to obtain the real-time health status of each storage node in real time, and based on the real-time health status of each storage node, assign the Shuffle data reading task or Shuffle data writing task transmitted by the client to the Shuffle data processing service module.
[0016] Optionally, the Shuffle processing service module includes:
[0017] A data merging submodule, configured to merge and process the first pre-processed data according to the preset data merging algorithm, and determine a target storage node for the first pre-processed data in the same key value range;
[0018] The metadata storage submodule is used to determine the metadata of each first preprocessed data after each first preprocessed data is stored in the corresponding target storage node, and send the metadata of the first preprocessed data to the Shuffle metadata management service module for storage.
[0019] Optionally, the Shuffle processing service module includes:
[0020] A data reading interface is used to interact with the Shuffle metadata management service module and the Shuffle data storage service module, obtain the metadata of the target first pre-processed data transmitted by the Shuffle metadata management service module, and forward the target first pre-processed data request message to the Shuffle data storage service module according to the metadata mapping relationship of the target first pre-processed data.
[0021] In a second aspect, the present application provides a data processing method based on the Shuffle process, the method being applied to the big data engine system described in the first aspect, the method comprising:
[0022] Receiving offline data analysis tasks, the offline data analysis tasks include: Shuffle data reading tasks and Shuffle data writing tasks;
[0023] Based on the key value of each first preprocessed data, a preset data merging algorithm is used to merge and process each first preprocessed data, wherein the first preprocessed data is: data located on different data shards obtained after the offline data analysis task Map calculation is completed; wherein the preset data merging algorithm is used to merge and map the first preprocessed data located in the same key value range to the same target storage node based on the key value of each first preprocessed data;
[0024] According to the key value of the target storage node to which each of the first pre-processed data belongs, each of the first pre-processed data is stored in the corresponding target storage node;
[0025] The metadata of each of the first pre-processed data is stored, where the metadata is storage index information of each of the first pre-processed data in each of the target storage nodes.
[0026] Optionally, the offline data analysis task further includes: a Shuffle data reading task, wherein the Shuffle data reading task carries target first pre-processed data request information to be read; the method further includes:
[0027] In response to the target first pre-processed data request information, delivering metadata of the target first pre-processed data to the Shuffle data storage service module;
[0028] Based on the metadata of the target first pre-processed data, the target first pre-processed data is acquired, and the target first pre-processed data is returned to the client.
[0029] Optionally, the method further includes:
[0030] The real-time health status of each storage node is obtained in real time, and based on the real-time health status of each storage node, the Shuffle data reading task or Shuffle data writing task transmitted by the client is assigned to the Shuffle data processing service module.
[0031] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes:
[0032] Processor; and
[0033] Memory for storing programs,
[0034] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes the data processing method based on the Shuffle process described in the second aspect.
[0035] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the data processing method based on the Shuffle process described in the second aspect.
[0036] Beneficial effects of this application:
[0037] The present application provides a big data engine system, a data processing method based on the Shuffle process, and an electronic device, wherein the big data engine system at least includes: a client, a Shuffle processing service module, a Shuffle data storage service module, and a Shuffle metadata management service module. The application embodiment is selected to receive an offline data analysis task, especially a Shuffle data writing task, through the client, and the first pre-processed data located on different data shards obtained by the offline data analysis task through Map calculation are merged and mapped to the same target storage node based on their own key values using a preset data merging algorithm, and then the first pre-processed data located in the same key value range are respectively stored in the corresponding target storage node according to the key value of the target storage node to which each first pre-processed data belongs, and the storage index information of the first pre-processed data in each target storage node is stored in the Shuffle metadata management service module.
[0038] By selecting the embodiment of the present application, since the first preprocessed data that originally needed to be stored in different data shards according to their own key values are merged and stored in fewer data shards according to the key value range to which the key values belong, when the subsequent big data engine system needs to obtain the first preprocessed data when performing the Shuffle read operation, the corresponding target first preprocessed data can be obtained from the fewer data shards, which effectively reduces the frequent access to the disk in the Shuffle stage, saves the time for obtaining data across target storage nodes in the Shuffle stage, reduces the IO occupied by the Shuffle stage, and saves the network resources and system resources occupied by the Shuffle stage, which can help to improve and ensure the data analysis efficiency of the big data computing engine. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0040] Figure 1 A schematic diagram of the architecture of a big data computing engine system provided in an embodiment of the present application is shown;
[0041] Figure 2 Another schematic diagram of the working principle of the Shuffle processing service module provided in an embodiment of the present application is shown;
[0042] Figure 3 Another schematic diagram of the architecture of the big data computing engine system provided in an embodiment of the present application is shown;
[0043] Figure 4 A schematic diagram showing a working principle of data merging of the Shuffle processing service module provided in an embodiment of the present application is shown;
[0044] Figure 5 Another schematic diagram of the architecture of the big data computing engine system provided in an embodiment of the present application is shown;
[0045] Figure 6 A schematic diagram of a process of a data processing method based on the Shuffle process provided in an embodiment of the present application is shown;
[0046] Figure 7 A schematic diagram of a logical architecture of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0047] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.
[0048] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.
[0049] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0050] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0051] The Shuffle phase is the core phase of the MapReduce data processing phase. It is mainly used to reorganize or redistribute data in the fields of computer science and big data processing, especially in the distributed computing framework. For example, the MapReduce process of Apache Hadoop (a distributed computing framework) can be divided into the Map phase, the Shuffle phase, and the Reduce phase. Among them, the Shuffle phase is located between the Map phase and the Reduce phase, and is used to reorganize or redistribute data for the next step of data sorting or data analysis.
[0052] The data shuffle process involves a large amount of data transmission across nodes, which can easily cause the data shuffle to occupy a large amount of processing time in the entire data computing process, becoming a major time-consuming process in the overall computing process. This not only consumes a large amount of network resources, but may also cause task delays and excessive occupation of system resources. Especially in a large-scale cluster environment, the efficiency of data shuffle directly affects the completion time of data analysis tasks, and thus affects the timeliness and accuracy of business decisions.
[0053] For example, the Map phase passes 9 Partition data to the Shuffle phase:
[0054] Partition a, Partition b, Partition c, Partition d, Partition e, Partition f, Partition g, Partition h, Partition i, the 9 Partition data belong to different storage nodes: node a, node b, node c, ..., node i. At this time, it is necessary to write the data of Partition a to Partition i to nodes a to i respectively. At this time, 9 data write operations need to be performed. When obtaining the data of Partition a to Partition i, it is necessary to perform 9 data read operations from nodes a to i. In the actual big data computing engine, the number of data read and write operations is massive. The technical means adopted in the existing Shuffle stage are prone to the problems of frequent local disk reading and writing, occupying a large number of IO interfaces, and consuming a large number of network resources and system resources, which further affects the efficiency of the big data computing engine in acquiring data and ultimately affects the analysis efficiency of the big data computing engine.
[0055] In this context, how to improve the efficiency of big data engines, especially the efficiency of data shuffle, has become the key to improving the timeliness and accuracy of business decisions. The commonly used methods for improving the efficiency of big data engines in the industry are:
[0056] Method 1: Big data engine efficiency improvement based on Alluxio. The integrated application of Alluxio in the big data processing engine has innovatively built a bridge that closely and efficiently connects computing and storage, achieving a significant leap in data access and processing efficiency. By caching hot data in memory, Alluxio cleverly utilizes the high-speed access characteristics of memory, greatly reducing I / O waiting time, making data access as smooth as flowing water, and accelerating the operation rhythm of data-intensive applications. It not only optimizes data locality and ensures that computing tasks are executed in the nearest place to the data, but also uses intelligent data preloading strategies, as if it has the ability to foresee the future, placing data that will be frequently accessed in the best position in advance, paving a high-speed channel for computing tasks.
[0057] Method 2: Performance optimization of the big data engine based on Lucene. The big data engine provides powerful distributed computing capabilities, while Lucene is a high-performance full-text search engine library. By combining the two, the big data engine can be used to process a large number of documents in a distributed manner, generate distributed indexes, and use Lucene for efficient search. This not only enables the processing of large-scale data sets, but also enables fast text retrieval in a distributed environment, greatly improving search efficiency. The combination with Lucene, through distributed processing, in-memory computing, real-time index updates, and resource optimization, not only enhances the performance of text searches, but also expands the depth and breadth of big data analysis, providing an efficient and flexible solution for processing large-scale text data and complex query requirements.
[0058] Method 3: Columnar storage and compression for big data storage. Columnar storage and compression technology in big data storage is a key strategy to improve data processing efficiency and optimize storage space. Columnar storage, different from traditional row storage, is a storage method that clusters data by column. In this mode, data in the same column is stored closely together, which is particularly beneficial for analytical queries because such queries usually only involve certain specific columns of the data set. A major advantage of columnar storage is its efficient data compression capability. Since the data type in the same column is consistent and there are a large number of duplicate values, the use of columnar storage can implement more efficient compression algorithms for these similar data, greatly reducing the demand for storage space.
[0059] In addition, columnar storage only needs to load the queried columns when reading, which reduces I / O operations and speeds up query processing, especially in analytical scenarios involving large amounts of data. Data compression is a technology that further optimizes storage and transmission efficiency. It processes raw data through specific algorithms, converting it into a form that takes up less space and can be restored to its original state when needed. In a big data environment, it is crucial to choose a compression algorithm appropriately. For read-intensive operations, you may prefer to use a fast decompression algorithm to reduce processing delays; for write-intensive or archival storage, you may choose an algorithm with a high compression ratio to maximize storage efficiency.
[0060] Although the above three methods can improve the efficiency of big data engines, they still have the following limitations:
[0061] Limitation 1: The shuffle computing efficiency problem is not really solved from the bottom of the engine, and the effect is not obvious.
[0062] Existing technologies have not fundamentally and thoroughly solved the problem of computing efficiency in the Shuffle stage. As a key step in the MapReduce computing model, Shuffle is responsible for redistributing intermediate data for Reduce task aggregation processing. The industry continues to explore such as deepening the integration of distributed memory management systems, smarter data distribution algorithms, and new computing models, in order to solve the Shuffle efficiency problem at a deeper level, reduce unnecessary data movement, and improve overall computing throughput and response speed, but the results cannot fully meet business expectations.
[0063] Limitation 2: Data migration and integration are costly. When migrating from traditional storage to columnar storage or introducing a new data cache layer (such as Alluxio), the cost of data migration is high and may affect the continuity of existing business. Compatibility issues between different storage formats, indexing technologies, and computing frameworks increase the complexity of system integration. At the same time, data migration and integration is a highly complex and risky process that requires cross-departmental collaboration, careful planning, sufficient technical preparation, and strict risk management, and the cost is very high.
[0064] Limitation 3: Limited applicable scenarios and lack of universality. Although the current common technical solutions have demonstrated high efficiency and practicality in specific fields or limited conditions, their targeted design and implementation inevitably limit their wide applicability and universality. The main reason is that common technical solutions are mainly aimed at environments with clear boundary requirements and specific technology stacks, and it is difficult to cover all aspects of diverse business scenarios and technical requirements. In particular, with the rapid development of big data, cloud computing, and artificial intelligence technologies, various application scenarios have increasingly higher requirements for flexibility, scalability, and compatibility, and the limitations of the current technical framework are gradually emerging. It may struggle when the data scale expands rapidly, or appear inflexible when highly customized integration is required, or show insufficient compatibility when facing emerging technology standards and protocols.
[0065] In view of this, the present application provides a big data engine system, a data processing method based on the Shuffle process, and an electronic device. In the first aspect, the present application provides a big data engine system, which can be pre-installed in any electronic device with big data processing capabilities, including but not limited to personal mobile terminals, computers or servers, etc. Figure 1 As shown, the system includes the following parts:
[0066] Client 11, Shuffle processing service module 12, Shuffle data storage service module 13, Shuffle metadata management service module 14. Among them:
[0067] The client 11 is used to receive offline data analysis tasks, which include: Shuffle data reading tasks and Shuffle data writing tasks;
[0068] The Shuffle processing service module 12 is used to merge and process each of the first pre-processed data based on the key value of each of the first pre-processed data using a preset data merging algorithm, wherein the first pre-processed data is: the data located on different data shards obtained after the offline data analysis task Map calculation is completed; wherein the preset data merging algorithm is used to merge and map the first pre-processed data located in the same key value range to the same target storage node based on the key value of each of the first pre-processed data;
[0069] The Shuffle data storage service module 13 is used to store each of the first pre-processed data to a corresponding target storage node according to the key value of the target storage node to which each of the first pre-processed data belongs;
[0070] The Shuffle metadata management service module 14 is used to store metadata of each of the first pre-processed data, where the metadata is storage index information of each of the first pre-processed data in each of the target storage nodes.
[0071] An application embodiment is selected, in which an offline data analysis task is received through a client, and the first preprocessed data located on different data shards obtained by Map calculation of the offline data analysis task is merged and mapped to the same target storage node based on its own key value using a preset data merging algorithm. Then, according to the key value of the target storage node to which each first preprocessed data belongs, each first preprocessed data is stored in the corresponding target storage node, and the storage index information of the first preprocessed data in each target storage node is stored in the Shuffle metadata management service module.
[0072] In order to facilitate understanding, Figure 2 The schematic diagram shown is used as an example: the traditional Shuffle stage processing of the first preprocessed data usually uses the key value of the first preprocessed data as the storage metadata (or storage index basis), and stores each first preprocessed data in the corresponding shard. Taking the above-mentioned Partition a to Partition i data as the first preprocessed data as an example, if the Partition a to Partition i data is to be written to the back-end storage nodes partition a to paritition i, the processing idea of the traditional Shuffle processing technology is: write each Partition data to its own storage node partition a to paritition i. At this time, when reading the data later, it is necessary to cross 9 storage nodes to obtain the Partition a to Partition i data.
[0073] By selecting the big data engine system provided by the embodiment of the present application, through the Shuffle processing service module provided by the embodiment of the present application, a total of 9 data items of Partition a to Partition i can be merged and mapped to the same target storage node according to a specific key value division method, and then the data belonging to the same key value range can be stored in the corresponding target storage node (or target storage shard) based on the merged mapping relationship. In this way, the 9 partition data items are divided into 4 storage nodes: partition1 to partition4, and then only 4 storage nodes need to be crossed to obtain the target first pre-processed data, which greatly reduces the frequent access to the disk in the Shuffle stage and saves the time of obtaining data across target storage nodes in the Shuffle stage.
[0074] It can be seen that by selecting the embodiment of the present application, since the first preprocessed data that originally needed to be stored in different data shards according to their own key values are merged and stored in fewer data shards according to the key value range to which the key values belong, when the subsequent big data engine system needs to obtain the first preprocessed data to perform data analysis tasks, the corresponding target first preprocessed data can be obtained from the fewer data shards, which effectively reduces the frequent access to the disk in the Shuffle stage, saves the time for obtaining data across target storage nodes in the Shuffle stage, reduces the IO occupied by the Shuffle stage, and saves the network resources and system resources occupied by the Shuffle stage, which can help to improve and ensure the data analysis efficiency of the big data computing engine.
[0075] The following is a detailed description of the big data computing engine system provided by this application in conjunction with the accompanying drawings of the specification:
[0076] In the implementation of this application, the big data computing engine can be a computing engine in various cloud systems, including computing engines in hybrid cloud, private cloud, dedicated cloud and other cloud systems. Among them, hybrid cloud integrates public cloud and private cloud, and is the main model and development direction of cloud computing in recent years. Among them, private cloud is for enterprise users. For security reasons, enterprises are more willing to store data in private cloud, but hope to obtain computing resources of public cloud. In this case, hybrid cloud is widely used. By mixing and matching public cloud and private cloud, it can achieve both security and cost-saving effects.
[0077] Since the key factors affecting the execution speed of the existing big data computing engine are the frequent random disk reads and high network overhead in the data shuffle process, the data processing efficiency of the entire big data computing engine is poor. The embodiment of the present application improves the processing logic of the shuffle stage in the existing big data computing engine, merges the intermediate data according to the Partition key value in the shuffle stage, and then uniformly sends the Partition data to the shuffle data storage service module 13, thereby reducing local disk read and write and network overhead.
[0078] As mentioned above, the MapReduce process is divided into the Map phase, the Shuffle phase, and the Reduce phase. The Map phase is mainly responsible for dividing the input data set according to the file size and location information, decomposing it into a series of key-value pairs, that is, dividing the data set into several data blocks of appropriate size, each of which is processed by a Mapper. Mapper is a component that executes data mapping and SQL statement definition, and is used to interact directly with the backend database, execute SQL statements, and return query results.
[0079] In an embodiment of the present application, in the offline data analysis tasks received by the client, especially in the Shuffle data writing tasks, there are data sets that need to be written. At this time, the data sets carried in the Shuffle data writing tasks are disassembled in the Map stage to obtain data blocks, and each data block corresponds to each first pre-processed data. Among them, after being processed in the Map stage, each first pre-processed data can obtain an initial storage location. Since the existing database systems mostly use distributed storage systems, the initial storage location obtained by each first pre-processed data corresponds to each data shard, that is, each first pre-processed data corresponds to each Partition data in the above example, and the key value of each first pre-processed data corresponds to each Partition key value. In an embodiment of the present application, the Partition key value is a mechanism for data partitioning. The technical principle of the Partition key value is: distribute data in different partitions according to the specified key value, thereby improving the performance and maintainability of the entire data system.
[0080] As an implementation method, the Shuffle data storage service module 13 is an important part of the entire big data computing engine system, which is used to ensure the persistence and accessibility of the output results of the Shuffle processing service module 12. Specifically, by persistently storing the data in HDFS (Hadoop Distributed File System), the HDFS storage mode provides a highly scalable and fault-tolerant storage solution for the entire data service. Under this HDFS storage mode, the entire big data computing engine system extracts key information through parsing and stores the serialized content in HDFS. Specifically, after receiving a write request, the Shuffle data storage service module 13 will call the HDFS.Write() method to write each first pre-processed data into the distributed data system, and verify the storage status of data blocks, copies, etc. Correspondingly, if the subsequent Shuffle data storage service module 13 receives a read request, the target first pre-processed data can be obtained by calling the HDFS.Read() method.
[0081] In some embodiments, the Shuffle processing service module is shown in 3, and specifically includes:
[0082] The data merging submodule 121 is used to merge the first pre-processed data according to the preset data merging algorithm, and determine the target storage node for the first pre-processed data in the same key value range;
[0083] The metadata storage submodule 122 is used to determine the metadata of each first preprocessed data after each first preprocessed data is stored in the corresponding target storage node, and send the metadata of the first preprocessed data to the Shuffle metadata management service module 14 for storage.
[0084] The data storage submodule 123 is used to store each first pre-processed data in the shuffle data storage service module 13.
[0085] Based on this, when the Shuffle processing service module merges the first preprocessed data, it specifically calls the internal data merging submodule 121 to implement it. The data merging submodule 121 merges the first preprocessed data according to the preset data merging algorithm, and determines the corresponding target storage node for the first preprocessed data in the same key value range.
[0086] In some embodiments, Figure 3 As shown, the offline data analysis task also includes: a Shuffle data reading task, wherein the Shuffle data reading task carries the target first pre-processed data request information to be read;
[0087] The Shuffle metadata management service module 14 is further configured to transmit the metadata of the target first pre-processed data to the Shuffle data storage service module in response to the target first pre-processed data request information;
[0088] The Shuffle data storage service module 13 is used to obtain the target first pre-processed data based on the metadata of the target first pre-processed data, and return the target first pre-processed data to the client.
[0089] As an implementation mode, the preset data merging algorithm can call the acquisition partitioning method Gethashpartition of the big data computing engine through the data merging submodule, and can merge the various first pre-processed data input in the Map stage.
[0090] The data merging submodule specifically performs a merging process on each first pre-processed data as follows: Figure 4 As shown:
[0091] The Shuffle data is passed into the Gethashpartition method as a data input parameter, and the Gethashpartition method calls the hash function to map the input Shuffle data to an integer within a fixed numerical range, and then distributes the data to the corresponding partition according to the partition index. Among them, the hash function can be an MD5 hash function, or a SHA-1 hash function, and the specific hash function type is not strictly limited in this article. Among them, the hash value can be converted into an integer through a modulus operation, so that Part1, Part2...PartN, a total of N partitions, one partition corresponds to a storage service node, and then the actual space situation of the storage service node is checked to determine whether the remaining space size meets the storage of the corresponding Part1 range of the Partition data. If it meets the requirements, the corresponding storage service node is selected, and each first pre-processed data is written to the Shuffle data storage service module 13 through the data storage submodule.
[0092] During the process, data belonging to the same partition key value range will be stored separately in a message queue for data merging, and then the actual storage situation of the Shuffle storage service will be detected. At this time, the Shuffle data storage service module 13 will return information such as the health status of each storage node, the remaining space, and the network topology diagram between the big data client. The Shuffle processing service module 12 can calculate the storage location of each Partition data through the storage location evaluation algorithm based on the data returned by the Shuffle data storage service module 13, and then send the Partition data to the Shuffle data storage service module 13 for data storage.
[0093] After the data storage is completed, the data will be checked for consistency to determine that the data has been accurately and uniquely stored in the Shuffle data storage service module 13. Then the storage service will send a successful ack signal to the Shuffle processing service. When the storage service receives the success signal, it will send the location information, data size and other information of each first pre-processed data to the metadata management center for storage. After the storage is completed, the entire Shuffle write process ends and a success signal is sent to the big data engine scheduler.
[0094] In the embodiment of the present application, the Shuffle processing service module is mainly responsible for the specific writing operation of the Shuffle data writing task and the specific data pulling operation of the Shuffle data reading task below. After the calculation of the Map phase of an offline task is completed, all the Partition data generated by it is sent to the Shuffle processing service module, and then the data merging submodule in the Shuffle processing service module merges all the data according to the key value of the Partition data, which can greatly reduce the number of data partitions. The data of the same Partition will then be sent to the same target storage node for storage. In this way, when pulling data in the subsequent Reduce phase, there is no need to perform interactive data pulling across multiple nodes, and the data of the same Partition can be pulled from the same storage node, which fundamentally greatly reduces the frequency of random disk reads.
[0095] In the embodiment of the present application, the client is a client of the big data engine Spark, Hive or Hadoop. During the entire offline task execution process, the engine scheduler will distinguish between data read operations and write operations according to the calculation situation of the Map stage, and then perform differential processing. Specifically, differential processing can be performed through the Shuffle task scheduling module. Based on this, in some embodiments, Figure 5 As shown, the system also includes:
[0096] The Shuffle task scheduling module 15 is used to obtain the real-time health status of each storage node in real time, and based on the real-time health status of each storage node, assign the Shuffle data reading task or Shuffle data writing task transmitted by the client to the Shuffle data processing service module.
[0097] In some embodiments, Figure 5 As shown, the Shuffle processing service module also includes:
[0098] The data reading interface 124 is used to interact with the Shuffle metadata management service module 14 and the Shuffle data storage service module 13, obtain the metadata of the target first pre-processed data transmitted by the Shuffle metadata management service module, and forward the target first pre-processed data request message to the Shuffle data storage service module according to the metadata mapping relationship of the target first pre-processed data.
[0099] Among them, the Shuffle task scheduling module is the data processing control center of the entire intermediate calculation results, which is responsible for the health check of the architecture and the allocation of data processing tasks. Among them, the Shuflle task scheduling module 15 can obtain the real-time health status of each storage node in real time, and can also periodically obtain the health status of each storage node according to a specific monitoring cycle. Exemplarily, information interaction can be performed every 20 seconds with the Shuffle processing service module, the Shuffle data storage service module, and the Shuffle metadata management service module to obtain the health status of other modules. If the detection fails for n consecutive times, the module will be shielded and no longer provide services to the outside. Among them, the value of n can be flexibly set according to actual experience. As a preferred implementation, n takes 3.
[0100] Among them, the health check mainly checks: the busyness of each processing module or storage node, the frequency of RPC (Remote Procedure Call) remote procedure call requests, the amount of data currently being processed or the amount of data to be processed, etc., to determine the health status of each processing module or storage node. Then, the feasibility assessment result of the write task is given so that the task allocation module can allocate tasks and ensure that the appropriate processing node is used for data processing.
[0101] In the embodiment of the present application, the specific response process of the Shuffle data writing task is as follows:
[0102] The engine scheduler in the client 11 will issue a Shuffle write operation command to the Shuffle task scheduling module 15. The Shuffle task scheduling module 15 calls the health status detection module 151 to detect the health status of all modules, and then calls the task allocation module 152 to uniformly allocate the Shuffle write task to the Shuffle processing service module 12 according to the health status detected by the health status detection module 151. The Shuffle processing service module 12 will merge the first pre-processed data with the same key value according to the received write task and map it to the same target storage node. The merged data is then sent to the Shuffle data storage service module 13 through the data storage submodule 123 for persistent storage.
[0103] After the Shuffle data storage service module 13 completes data storage, the Shuffle processing service module sends metadata information such as the storage node location and size of the storage data allocation to the Shuffle metadata management service module 14 so that the data information can be identified during subsequent Shuffle read operations.
[0104] Similarly, for the Shuffle data reading task, the engine scheduler in the client 11 will issue a Shuffle read operation command to the Shuffle task scheduling module 15. The Shuffle task scheduling module will first read the data information (the read data information carries the metadata of the target first preprocessed data), and then obtain the location information of the target first preprocessed data in the Shuffle data storage service module 13 according to the metadata of the target first preprocessed data, and then assign the data reading task to the Shuffle processing service module 12 through the task assignment module 152. The Shuffle processing service module 12 interacts with the Shuffle data storage service module 13 from the data reading interface 124 to obtain the corresponding data, and then sends it to the client to continue data processing in the Reduce stage.
[0105] In the second aspect, the present application also provides a data processing method based on the Shuffle process, which is mainly applied to the big data engine system provided in the first aspect, as shown in FIG. Figure 6 As shown, the method comprises the following steps:
[0106] S61, receiving an offline data analysis task, wherein the offline data analysis task at least includes: a Shuffle data reading task and a Shuffle data writing task;
[0107] S62, based on the key value of each first pre-processed data, using a preset data merging algorithm to merge the first pre-processed data, wherein the first pre-processed data is: the data located on different data shards obtained after the offline data analysis task Map calculation is completed; wherein the preset data merging algorithm is used to merge and map the first pre-processed data located in the same key value range to the same target storage node based on the key value of each first pre-processed data;
[0108] S63: storing each of the first preprocessed data in a corresponding target storage node according to the key value of the target storage node to which each of the first preprocessed data belongs;
[0109] S64: Store metadata of each of the first pre-processed data, where the metadata is storage index information of each of the first pre-processed data in each of the target storage nodes.
[0110] The specific implementation process of each step can refer to the corresponding description content in the first aspect, which will not be repeated here. In an embodiment of the present application, a distributed data shuffle processing method independent of the storage node is proposed, which can centrally manage and process the intermediate data generated during the data shuffle process, thereby reducing the direct transmission of data between different storage nodes, which is conducive to reducing network congestion, thereby improving the efficiency and stability of data processing.
[0111] By selecting the embodiment of the present application, the Shuffle operation is abstracted into an independent service layer to achieve decoupling of computing and data exchange, which can effectively solve the performance bottleneck problem of the Shuffle stage in data-intensive tasks, thereby improving the data processing efficiency and accuracy of the big data computing engine.
[0112] In some embodiments, the offline data analysis task further includes: a Shuffle data reading task, wherein the Shuffle data reading task carries the target first pre-processed data request information to be read; the method further includes:
[0113] In response to the target first pre-processed data request information, delivering metadata of the target first pre-processed data to the Shuffle data storage service module;
[0114] Based on the metadata of the target first pre-processed data, the target first pre-processed data is acquired, and the target first pre-processed data is returned to the client.
[0115] In some embodiments, the method further comprises:
[0116] The real-time health status of each storage node is obtained in real time, and based on the real-time health status of each storage node, the Shuffle data reading task or Shuffle data writing task transmitted by the client is assigned to the Shuffle data processing service module.
[0117] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.
[0118] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0119] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.
[0120] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.
[0121] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.
[0122] refer to Figure 7 , the structural block diagram of the electronic device 700 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.
[0123] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0124] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0125] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the aforementioned data processing method based on the Shuffle process may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the aforementioned data processing method based on the Shuffle process in any other appropriate manner (e.g., by means of firmware).
[0126] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0127] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0128] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0130] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0131] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
Claims
1. A big data engine system, characterized in that: The big data engine system at least includes: The client is used to receive offline data analysis tasks, wherein the offline data analysis tasks include: Shuffle data reading tasks and Shuffle data writing tasks; The Shuffle processing service module is used to merge and process each of the first pre-processed data based on the key value of each of the first pre-processed data using a preset data merging algorithm, wherein the first pre-processed data is: the data located on different data shards obtained after the offline data analysis task Map calculation is completed; wherein the preset data merging algorithm is used to merge and map the first pre-processed data located in the same key value range to the same target storage node based on the key value of each of the first pre-processed data; A Shuffle data storage service module, configured to store each of the first pre-processed data in a corresponding target storage node according to a key value of the target storage node to which each of the first pre-processed data belongs; The Shuffle metadata management service module is used to store metadata of each of the first pre-processed data, where the metadata is storage index information of each of the first pre-processed data in each of the target storage nodes.
2. The system according to claim 1, characterized in that The Shuffle data reading task carries the target first pre-processed data request information to be read; The Shuffle metadata management service module is further used to transmit the metadata of the target first pre-processed data to the Shuffle data storage service module in response to the target first pre-processed data request information; The Shuffle data storage service module is used to obtain the target first preprocessed data based on the metadata of the target first preprocessed data, and return the target first preprocessed data to the client.
3. The system according to claim 1, characterized in that The system further comprises: The Shuffle task scheduling module is used to obtain the real-time health status of each storage node in real time, and based on the real-time health status of each storage node, assign the Shuffle data reading task or Shuffle data writing task transmitted by the client to the Shuffle data processing service module.
4. The system according to claim 1, characterized in that The Shuffle processing service module includes: A data merging submodule, configured to merge and process the first pre-processed data according to the preset data merging algorithm, and determine a target storage node for the first pre-processed data in the same key value range; The metadata storage submodule is used to determine the metadata of each first preprocessed data after each first preprocessed data is stored in the corresponding target storage node, and send the metadata of the first preprocessed data to the Shuffle metadata management service module for storage.
5. The system according to claim 2, characterized in that The Shuffle processing service module includes: A data reading interface is used to interact with the Shuffle metadata management service module and the Shuffle data storage service module, obtain the metadata of the target first pre-processed data transmitted by the Shuffle metadata management service module, and forward the target first pre-processed data request message to the Shuffle data storage service module according to the metadata mapping relationship of the target first pre-processed data.
6. A data processing method based on the Shuffle process, characterized in that: The method is applied to the big data engine system according to any one of claims 1 to 6, and the method comprises: Receiving an offline data analysis task, wherein the offline data analysis task at least includes: a Shuffle data writing task; Based on the key value of each first preprocessed data, a preset data merging algorithm is used to merge and process each first preprocessed data, wherein the first preprocessed data is: data located on different data shards obtained after the offline data analysis task Map calculation is completed; wherein the preset data merging algorithm is used to merge and map the first preprocessed data located in the same key value range to the same target storage node based on the key value of each first preprocessed data; According to the key value of the target storage node to which each of the first pre-processed data belongs, each of the first pre-processed data is stored in the corresponding target storage node; The metadata of each of the first pre-processed data is stored, where the metadata is storage index information of each of the first pre-processed data in each of the target storage nodes.
7. The method according to claim 6, characterized in that The offline data analysis task further includes: a Shuffle data reading task, wherein the Shuffle data reading task carries the target first pre-processed data request information to be read; the method further includes: In response to the target first pre-processed data request information, delivering metadata of the target first pre-processed data to the Shuffle data storage service module; Based on the metadata of the target first pre-processed data, the target first pre-processed data is acquired, and the target first pre-processed data is returned to the client.
8. The method according to claim 6, characterized in that The method further comprises: The real-time health status of each storage node is obtained in real time, and based on the real-time health status of each storage node, the Shuffle data reading task or Shuffle data writing task transmitted by the client is assigned to the Shuffle data processing service module.
9. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 6 to 8.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 6-8.
