A data reading and writing method, system and storage medium for data shuffling and transmission process of Spark application

Abstract

The present invention provides a data reading and writing method, system and storage medium for a data shuffling transmission process of a Spark application, the method comprising: in the writing stage, a memory writing module sends preliminary aggregated data in batches to a memory storage module corresponding to the writing stage; a memory pool unit allocates memory blocks to store the data of each batch sent by the memory writing module, a directory unit generates a memory block ID, records the corresponding relationship between the memory block ID and the memory block, and uploads the memory block ID to the memory writing module; in the reading stage, a communication connection is established with a memory server through a memory client to pull the preliminary aggregated data stored in the memory storage module corresponding to the writing stage, and the pulled data is stored in the memory storage module corresponding to the reading stage; a memory reading module reads the data in the memory storage module corresponding to the reading stage. The present invention can avoid reading and writing data from a disk when shuffling data, thereby greatly reducing the execution time of data shuffling transmission.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a data reading and writing method and system in a data shuffling and transmission process of a Spark application. Background Art

[0002] Spark is an open source big data processing framework that can be used for high-speed, easy-to-use, and highly flexible big data processing. Data shuffling is a key step in Spark application data processing. When certain operations (such as data grouping and sorting) need to be performed, data may need to be redistributed between different nodes in the Spark cluster. This process of redistributing data is called "Shuffle".

[0003] The data shuffling and transmission process of Spark applications in the prior art is as follows: Figure 1 As shown in the figure, the disk storage modules corresponding to the writing stage and the reading stage are all located in the disk, and the remaining modules (i.e., the disk writing module, the disk server, the disk client, and the disk reading module) are located in the memory. The arrows indicate the data flow. When a Spark task generates data that needs to be shuffled (e.g., a Map task), the disk writing module converts the original data into preliminary aggregated data and stores these preliminary aggregated data in a specific file directory unit on the disk (i.e., stored in the disk storage module corresponding to the writing stage), which is usually a subdirectory unit in the working directory unit of the Spark application. In addition, the Spark application will also generate an index file for quickly locating the location of specific data for subsequent read operations. In the reading stage (e.g., a Reduce task), the Spark task needs to read data from multiple nodes, quickly find the location of the required data according to the index file, pull the data through the disk client and the disk server, and then the disk reading module reads the data from the local disk or the disk of the remote node (the disk storage module corresponding to the reading stage). Since the data is stored on the disk, the disk data needs to be read frequently, and the disk has a slow reading and writing speed, which easily leads to performance bottlenecks.

[0004] Therefore, there is an urgent need for a shuffle process method for Spark applications with a new data reading and writing method to solve the performance bottleneck caused by the slow disk reading and writing speed. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data reading and writing method and system for the data shuffle transmission process of a Spark application, so as to avoid the performance bottleneck caused by disk reading and writing, thereby greatly reducing the execution time of the Shuffle and improving the efficiency of data processing in the Shuffle process.

[0006] One aspect of the present invention provides a data reading and writing method for a data shuffling transmission process of a Spark application, wherein the data shuffling transmission process includes a writing phase and a reading phase, and the method includes the following steps:

[0007] In the writing phase, the memory writing module obtains data, converts it into preliminary aggregated data, and sends the preliminary aggregated data in batches to the memory storage module corresponding to the writing phase; wherein the memory storage module corresponding to the writing phase includes a directory unit and a memory pool unit;

[0008] The memory pool unit allocates memory blocks to store the preliminary aggregated data of each batch sent by the memory writing module, the directory unit generates a memory block ID, records the correspondence between the memory block ID and the memory block, and uploads the memory block ID of each batch of data to the memory writing module; until all Spark tasks in the writing phase are executed; wherein the memory writing module includes a pre-stored memory server address, and each memory block ID of the preliminary aggregated data of each batch has a corresponding memory server address;

[0009] In the reading phase, the memory reading module obtains the memory block ID and the corresponding memory server address of each batch of data based on the memory writing module, and sends the data pull request, the obtained memory block ID and the obtained memory server address to the reading memory client;

[0010] The memory client establishes a communication connection with the memory server corresponding to the write phase based on the data pull request, memory block ID and memory server address issued by the memory read module to pull the preliminary aggregated data stored in the memory storage module corresponding to the write phase, and the memory client stores the pulled preliminary aggregated data in the memory storage module corresponding to the read phase;

[0011] The memory reading module reads the preliminary aggregation data stored in the memory storage module corresponding to the reading phase, and outputs the secondary aggregation data until all Spark tasks in the reading phase are executed.

[0012] In some embodiments of the present invention, before sending the preliminary aggregated data in batches to the memory storage module corresponding to the writing stage, it also includes: determining the storage capacity of the memory storage module corresponding to the writing stage, and when the storage capacity is less than a set threshold, writing the preliminary aggregated data sent by the memory writing module to the disk.

[0013] In some embodiments of the present invention, after the reading phase ends, the method further includes: deleting the preliminary aggregated data stored in the memory storage module corresponding to the writing phase;

[0014] The preliminary aggregated data stored in the memory storage module corresponding to the write phase is deleted in the following way:

[0015] Positioning step: The directory unit locates the corresponding memory block of the memory pool unit based on the correspondence between the recorded memory block ID and the memory block;

[0016] Removal step: deleting the preliminary aggregated data stored in the memory block corresponding to the memory pool unit, and removing the corresponding relationship between the memory block ID and the memory block from the directory unit;

[0017] The positioning step and the removal step are repeated until all preliminary aggregated data stored in the memory storage module corresponding to the written stage are deleted.

[0018] In some embodiments of the present invention, the memory writing module obtains data and converts it into preliminary aggregated data, including:

[0019] The memory writing module obtains raw data from the native Spark application through the first interface and converts the raw data into preliminary aggregated data; or

[0020] The memory writing module obtains preliminary aggregated data from the native Spark application through the first interface;

[0021] The memory reading module reads the preliminary aggregated data stored in the memory storage module corresponding to the reading phase, and outputs secondary aggregated data, including:

[0022] The memory reading module reads the preliminary aggregation data stored in the memory storage module corresponding to the reading phase, and outputs the secondary aggregation data to the native Spark application through the second interface.

[0023] In some embodiments of the present invention, the memory client establishes a communication connection with the memory server corresponding to the write phase based on the data pull request, the memory block ID and the memory server address issued by the memory read module to pull the preliminary aggregated data stored in the memory storage module corresponding to the write phase, including:

[0024] The memory client receives the data pull request, memory block ID and memory server address issued by the memory read module, and sends the data pull request and memory block ID to the specified memory server based on the memory server address;

[0025] The memory server receives the data pull request and locates the corresponding memory block in the memory storage module corresponding to the write phase based on the acquired memory block ID, so that the preliminary aggregated data stored in the corresponding memory block can be pulled by the corresponding memory client in the read phase.

[0026] In some embodiments of the present invention, locating a corresponding memory block in a memory storage module corresponding to the write phase based on the acquired memory block ID includes:

[0027] The memory server sends the acquired memory block ID to the memory storage module corresponding to the write phase;

[0028] The memory storage module corresponding to the write phase locates the memory block corresponding to the memory pool unit through the correspondence between the memory block ID recorded in the directory unit and the memory block and the acquired memory block ID.

[0029] In some embodiments of the present invention, the memory server is created using the Netty framework.

[0030] In some embodiments of the present invention, before sending a data pull request, an acquired memory block ID, and an acquired memory server address to a memory client for reading, it also includes: a memory reading module determines whether the memory server address is consistent with a local address in the memory client; if the memory server address is consistent with the local address, the memory reading module reads data from a memory storage module corresponding to the reading phase; if the memory server address is inconsistent with the local address, the memory reading module sends a data pull request to the memory client.

[0031] Another aspect of the present invention provides a data reading and writing system for the data shuffling transmission process of a Spark application, the system comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the method described in any of the above embodiments.

[0032] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in any of the above embodiments when the program is executed by a processor.

[0033] The data reading and writing method and system of the data shuffle transmission process of the Spark application of the present invention can avoid the performance bottleneck of disk I / O by directly storing the intermediate data generated by the Spark application in the memory, thereby greatly reducing the execution time of the Shuffle.

[0034] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0035] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0037] Figure 1 The figure is a schematic diagram of the data shuffling transmission process of the Spark application in the prior art according to an embodiment of the present invention.

[0038] Figure 2 The figure is a schematic diagram of the process of shuffling data in a native Spark application in one embodiment of the present invention.

[0039] Figure 3 The figure is a schematic diagram of the data shuffling transmission process of the Spark application in one embodiment of the present invention.

[0040] Figure 4 The figure is a flow chart of a data reading and writing method in a data shuffling and transmission process of a Spark application in one embodiment of the present invention.

[0041] Figure 5 Schematic diagram of the structure of a memory storage module corresponding to the reading phase in an embodiment of the present invention.

[0042] Figure 6 The figure is a schematic diagram of the data shuffling and transmission process of non-intrusive loading in a Spark application in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0044] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0045] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0046] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0047] In the present invention, Mapper can be represented as the actual executor of the writing phase (hereinafter referred to as Map phase), and similarly, Reducer can be represented as the actual executor of the reading phase (hereinafter referred to as Reduce phase).

[0048] The data reorganization and distribution process of Shuffle in native Spark applications is as follows Figure 2 shown. Figure 2 Different colors are used to represent different data, and the data in data block 1, data block 2, and data block 3 are the same. Data block 1 represents a data block obtained by the native Spark application before the Shuffle calculation. It is the original data, and these data are stored on different nodes of the cluster. Assuming that the goal of the Shuffle process is to group the data in data block 1 according to color, the Shuffle process will start. In the Map phase, the data on each node will be locally classified according to color to obtain data block 2. These classified data are called Shuffle intermediate data or preliminary aggregated data, and the native Spark application will temporarily store the intermediate data in the form of files on the disk of each node. In the Reduce phase, all data of the same color can be pulled through the network to be transmitted to different specified nodes to obtain data block 3. The data in data block 3 is called secondary aggregated data. At this point, the Spark application has completed the goal of grouping these data blocks according to color, and the Shuffle process ends. From the above cases, we can see that since the native Spark application does not use memory for data storage management during the Shuffle process, when writing data in the Map phase and reading data in the Reduce phase, it is necessary to write from memory to disk and then read from disk to memory, which greatly affects the performance of disk I / O. At the same time, although there are many implementations of Spark applications, they all try to group key-value pairs with the same key into a group to achieve a relatively even grouping effect. In addition, although Spark applications support code modification, there are also problems such as code failure after modification and limited support scope.

[0049] Based on this, an embodiment of the present invention provides a data reading and writing method for the data shuffle transmission process (i.e., Shuffle process) of a Spark application, which uses memory instead of disk as the storage medium for Shuffle intermediate data, and introduces the relationship between memory block ID (which can be represented by buffer id) and memory block (which can be represented as buffer), and directly stores the Shuffle intermediate data in the local memory, so that the execution speed of the entire Shuffle process is significantly improved. In addition, the present invention adopts a non-invasive improvement method, designs corresponding storage modules, transmission modules, and management modules to be loaded from the outside of the Spark application, replaces the original storage modules, transmission modules, and management modules of the Spark application, and can seamlessly integrate the designed modules into the Spark application running environment through the native interface of the Spark application.

[0050] The modules involved in the data reading and writing method of the Shuffle process of the Spark application of the present invention can be divided into three categories. The memory writing module (which can be represented as ShuffleMemoryWriter) and the memory reading module (which can be represented as ShuffleMemoryReader) can be used as management modules, respectively used to write data and read data; the memory storage module corresponding to the writing phase (which can be referred to as Memory Storage on the Mapper side) and the memory storage module corresponding to the reading phase (which can be referred to as Memory Storage on the Reducer side) can be used as storage modules for storing preliminary aggregated data; the memory server (which can be represented by Memory Server) and the memory client (which can be represented by Memory Client) can be used as transmission modules for transmitting preliminary aggregated data. The Shuffle process of the Spark application of the present invention is as follows: Figure 3 shown. Figure 3 All modules in the diagram are located in memory, and arrows indicate the direction of data flow. In the write phase, the ShuffleMemoryWriter on the Mapper side obtains preliminary aggregated data and stores it in the local memory, that is, the Memory Storage on the Mapper side; in the read phase, data is pulled through the Memory Client and Memory Server, and then the ShuffleMemoryReader on the Reducer side reads data from the local memory (that is, the Memory Storage on the Reducer side).

[0051] Figure 4 The flowchart of the data reading and writing method of the shuffle process of the Spark application according to an embodiment of the present invention is as follows. Figure 4 The method shown may specifically include steps S110 to S150.

[0052] Step S110: In the Map phase, the ShuffleMemoryWriter on the Mapper side obtains data, converts it into preliminary aggregated data, and sends the preliminary aggregated data to the Memory Storage on the Mapper side in batches; wherein the Memory Storage on the Mapper side includes a directory unit and a memory pool unit. The memory pool unit can be divided into multiple memory blocks.

[0053] More specifically, the ShuffleMemoryWriter on the Mapper side is a module that has an interface with the native Spark application, and can connect and transmit data with the native Spark application through the first interface. The data transmitted between the native Spark application and the ShuffleMemoryWriter can be preliminary aggregated data or original data. If the transmitted data is preliminary aggregated data (i.e., the ShuffleMemoryWriter obtains preliminary aggregated data from the native Spark application), the ShufflerMemoryWriter acts as a data forwarding transit station and sends the obtained preliminary aggregated data to the Memory Storage on the Mapper side; if the transmitted data is original data (i.e., the ShuffleMemoryWriter obtains the original data from the native Spark application), the ShufflerMemoryWriter first partitions to obtain preliminary aggregated data, and then sends the preliminary aggregated data to the Memory Storage on the Mapper side. In addition, the preliminary aggregated data can be transmitted to the Memory Storage on the Mapper side in multiple batches, or multiple batches of data can be packaged and transmitted together to the Memory Storage on the Mapper side.

[0054] As an example, a native Spark application can pass data into ShufflerMemoryWriter in batches.

[0055] Furthermore, ShuffleMemoryWriter writes the preliminary aggregated data of each batch into the local Memory Storage (i.e., Memory Storage on the Mapper side) in batches, that is, ShufflerMemoryWriter writes the intermediate data into the memory.

[0056] Step S120: The memory pool unit allocates buffers to store the preliminary aggregated data of each batch sent by ShuffleMemoryWriter, the directory unit generates buffer ids, records the corresponding relationship between buffer ids and buffers, and uploads the buffer ids of each batch of data to ShuffleMemoryWriter; wherein ShuffleMemoryWriter includes a pre-stored memory server address (which can be represented by a Memory server address), and each batch of data has a corresponding Memory server address. If the determination module of the Spark application determines that all Spark tasks in the Map phase have been executed, the Map phase ends and the Reduce phase runs.

[0057] More specifically, the Memory Storage on the Mapper side is a module that can perform memory storage, and the Memory Storage on the Mapper side will allocate a globally unique buffer id for each batch of preliminary aggregated data sent by the ShufflerMemoryWriter. Figure 5 As shown in the figure, the Memory Storage on the Mapper side can be divided into two parts: the directory unit part (the white part on the left) and the memory pool unit part (the shaded part on the right). The directory unit is used to generate a globally unique memory block ID (buffer id) and record the correspondence between the memory block ID and the memory block (buffer); the memory pool unit is used to allocate buffers and manage memory, and the preliminary aggregated data of each batch is allocated in different buffers of the memory pool unit for storage. In addition, the directory unit can also be used to upload the buffer id of each batch of data to ShuffleMemoryWriter.

[0058] As an example, when the intermediate data is sent from ShuffleMemoryWriter to the local Memory Storage (i.e., the Memory Storage on the Mapper side), the Memory Storage on the Mapper side executes the data writing process; when the data stored in the Memory Storage on the Mapper side is pulled, the Memory Storage on the Mapper side executes the data reading process; when all Spark tasks in the Reduce phase are executed, the Memory Storage on the Mapper side executes the data deletion process. The data writing process on the Memory Storage on the Mapper side includes the following steps:

[0059] Step S01: Generate a globally unique buffer id from the directory unit, and allocate a buffer from the memory pool unit;

[0060] Step S02: the directory unit records the relationship between the newly generated buffer id and the newly allocated buffer;

[0061] Step S03: The preliminary aggregated data of the current batch is written into the newly allocated buffer.

[0062] As an example, the buffer may be allocated randomly or in a certain manner, but the present invention is not limited thereto. For example, the buffer may be allocated based on an operating system.

[0063] Furthermore, ShuffleMemoryWriter pre-stores all the Memory server addresses on the Mapper side, and each allocated buffer of the memory pool unit (or each batch of preliminary aggregated data stored corresponding to each buffer id) has a corresponding Memory server address. Therefore, the preliminary aggregated data of the memory pool unit stored in the Memory Storage on the Mapper side has a unique corresponding buffer id and Memory server address.

[0064] As an example, ShuffleMemoryWriter applies to the directory unit in the Memory Storage on the Mapper side to store preliminary aggregate data and transfers the data to the directory unit. The directory unit generates a buffer id for the preliminary aggregate data to be stored, and the directory unit stores the preliminary aggregate data in the buffer allocated by the memory pool unit. After the data is stored, the directory unit records the newly generated buffer id and the corresponding buffer, and uploads the newly generated buffer id to ShuffleMemoryWriter. ShuffleMemoryWriter packages and compresses all the buffer ids in a Spark task, and reports them to the Shuffle tracking module (the Shuffle tracking module is a built-in component of the Spark application, which can be represented as MapOutputTracker) together with the pre-stored Memory server address, so that MapOutputTracker holds the corresponding information of the buffer id and the Memory server address.

[0065] As an example, MapOutputTracker may represent the correspondence between the buffer ID and the Memoryserver address in the form of a key-value pair, or in the form of a hash table, but the present invention is not limited thereto.

[0066] Furthermore, as a built-in component of the native Spark application, the determination module can be used to determine whether all Spark application tasks in the Map phase have been executed, and can also be used to determine whether all Spark tasks in the Reduce phase have been executed. If the determination module determines that all Spark tasks in the current phase have been executed, it proceeds to the next phase. For example, after the Map phase is completed, the Reducer starts running the Reduce phase of the Shuffle process.

[0067] Step S130: In the Reduce phase, the ShuffleMemoryReader on the Reducer side obtains the buffer id and the corresponding Memory server address of each batch of data based on the ShuffleMemoryWriter, and sends the data pull request and the obtained buffer id and Memory server address to the MemoryClient on the Reducer side.

[0068] As an example, ShuffleMemoryReader can obtain the buffer id and the corresponding Memory server address from MapOutputTracker, and send a data pull request, the obtained buffer id and the obtained Memory server address to the Memory Client on the Reducer side.

[0069] More specifically, ShuffleMemoryReader is a module that interfaces with native Spark applications. When Reducer obtains the buffer id and the MemoryServer address where the data is located from the built-in component MapOutputTracker of the Spark application, Reducer instantiates ShuffleMemoryReader and passes this information to ShuffleMemoryReader. Furthermore, after the ShuffleMemoryReader inside Reducer obtains the buffer id corresponding to the data stored in the Memory Storage on the Mapper side and the Server address where the data is located, it can pull data from the Memory Storage on the Mapper side through the Memory Client.

[0070] In some embodiments of the present invention, before sending the data pull request and the obtained buffer id and Memoryserver address to the Memory Client on the Reducer side, it also includes: ShuffleMemoryReader determines whether the Memory Server address is consistent with the local address in the Memory Client; if the Memory Server address is consistent with the local address, ShuffleMemoryReader reads data from the Memory Storage on the Reducer side; if the Memory Server address is inconsistent with the local address, ShuffleMemoryReader sends the data pull request to the Memory Client.

[0071] Furthermore, since the Memory Server address is globally unique, ShuffleMemoryReader can determine whether the Memory Server address is consistent with the local address (that is, the Memory Server address of the data to be pulled recorded in the Memory Client), that is, whether the data that ShuffleMemoryReader requires to be pulled is stored in the local Memory Storage (that is, the Memory Storage on the Reducer side). If the Memory Server address is consistent with the local address, ShuffleMemoryReader reads directly from the Memory Storage on the Reducer side; otherwise, ShuffleMemoryReader generates a Memory Client, sends a data pull request to the Memory Client, and pulls the required data by establishing a connection between the Memory Client and the corresponding Memory Server.

[0072] Step S140: Memory Client establishes a communication connection with the Memory server on the Mapper side based on the data pull request, buffer id and Memory server address issued by ShuffleMemoryReader to pull the preliminary aggregated data stored in the Memory Storage on the Mapper side, and Memory Client stores the pulled preliminary aggregated data in the Memory Storage on the Reducer side.

[0073] In some embodiments of the present invention, the Memory Client establishes a communication connection with the Memory server on the Mapper side based on the data pull request, buffer id and Memory server address issued by the ShuffleMemoryReader to pull the preliminary aggregated data stored in the Memory Storage on the Mapper side, including:

[0074] Memory Client receives the data pull request, buffer ID and Memory server address issued by ShuffleMemoryReader, and sends the data pull request and buffer ID to the specified Memory server based on the Memory server address;

[0075] The memory server receives the data pull request and locates the corresponding buffer in the MemoryStorage on the Mapper side based on the buffer id, so that the preliminary aggregated data stored in the corresponding buffer can be pulled by the MemoryClient on the Reducer side.

[0076] More specifically, after receiving the data pull request, buffer id and memory server address sent by ShuffleMemoryReader, Memory Client first establishes a communication connection with the memory server on the specific mapper side through the memory server address, and then Memory Client sends the request to pull data and the buffer id to the specific memory server. Once the specific memory server receives the request to pull data sent by the memory client, the specific memory server will pass the preliminary aggregated data stored in the memory storage on the mapper side to the memory client on the reducer side through the network based on the buffer id, and the memory client will store the pulled data in the memory storage on the reducer side.

[0077] In some embodiments of the present invention, locating the corresponding buffer in the Memory Storage of the Mapper side based on the buffer id includes:

[0078] The memory server sends the received buffer id to the memory storage on the mapper side;

[0079] The Memory Storage on the Mapper side locates the buffer corresponding to the memory pool unit through the correspondence between the buffer id and the buffer recorded in the directory unit and the buffer id.

[0080] Furthermore, the Memory Storage on the Mapper side sends the preliminary aggregated data and the corresponding buffer id required by the Memory Client to the Memory Client. The Memory Client records the buffer id that has been transmitted and the Memory server address corresponding to the buffer id (i.e., the local address mentioned in the present invention), and sends the transmitted preliminary aggregated data to the Memory Storage on the Reducer side for storage.

[0081] As an example, when the preliminary aggregated data stored in the Memory Storage on the Mapper side is pulled, the Memory Storage on the Mapper side performs a data reading process. The data reading process of the Memory Storage on the Mapper side includes the following steps:

[0082] S11: The directory unit locates the buffer corresponding to the memory pool unit through the buffer ID;

[0083] S12: The preliminary aggregated data of the batch is read out from the buffer corresponding to the memory pool unit.

[0084] As an example, Memory Server is created using the Netty framework.

[0085] Step S150: ShuffleMemoryReader reads the preliminary aggregated data stored in the Memory Storage of the Reducer, and outputs the secondary aggregated data. If all Spark tasks of the Reduce phase are completed, the Reduce phase ends.

[0086] In some embodiments of the present invention, the ShuffleMemoryWriter on the Mapper side obtains preliminary aggregated data, including:

[0087] The ShuffleMemoryWriter obtains raw data from the native Spark application through the first interface and converts the raw data into preliminary aggregated data; or, the ShuffleMemoryWriter obtains preliminary aggregated data from the native Spark application through the first interface.

[0088] ShuffleMemoryReader reads the preliminary aggregated data stored in the Memory Storage on the Reducer side and outputs secondary aggregated data, including:

[0089] The ShuffleMemoryReader reads the preliminary aggregated data stored in the Memory Storage on the Reducer side, and outputs the secondary aggregated data to the native Spark application through the second interface.

[0090] More specifically, ShuffleMemoryReader reads data from the Memory Storage on the Reducer side, that is, ShuffleMemoryReader reads intermediate data from the memory. When all Reducer tasks that rely on the initial aggregated data stored are successfully completed, the Reduce phase of Shuffle also ends.

[0091] In some embodiments of the present invention, after step S150, step S160 is further included: after the Reduce phase ends, the step further includes: deleting the preliminary aggregated data stored in the Memory Storage of the Mapper end;

[0092] The preliminary aggregated data stored in the Memory Storage on the Mapper side is deleted in the following way:

[0093] Positioning step: The directory unit locates the corresponding buffer of the memory pool unit based on the correspondence between the recorded buffer ID and the buffer;

[0094] Removal step: Delete the preliminary aggregate data stored in the buffer corresponding to the memory pool unit, and remove the corresponding relationship between the buffer ID and the buffer from the directory unit;

[0095] Repeat the positioning and removal steps until all preliminary aggregation data stored in the Memory Storage on the Mapper side is deleted.

[0096] As an example, when the preliminary aggregate data stored in the Memory Storage on the Mapper side is to be deleted, all the preliminary aggregate data stored in the Memory Storage on the Mapper side is pulled by the Memory Client, and all the pulled data stored in the Memory Storage on the Reducer side is read by the ShuffleMemoryReader.

[0097] In some embodiments of the present invention, before sending the preliminary aggregated data in batches to the memory storage module corresponding to the writing stage, it also includes: determining the storage capacity of the Memory Storageag on the Mapper side, and when the storage capacity is less than the set threshold, writing the preliminary aggregated data sent by the ShuffleMemoryWriter into the shuffle file of the disk, that is, writing it into the storage module of the disk.

[0098] As an example, when the storage capacity of the Memory Storage on the Mapper side is small, the data sent by ShuffleMemoryWriter will be stored in the shuffle file on the disk according to the data storage path of the native Spark application.

[0099] In a specific embodiment of the present invention, a data reading and writing device for a shuffle process of a Spark application includes:

[0100] ShuffleMemoryWriter on the Mapper side is used to obtain the buffer id generated by the preliminary aggregation data and directory unit, and transmit the buffer id and the pre-stored memory server address to the built-in component MapOutputTracker of the Spark application;

[0101] Memory Storage on the Mapper side is used to store preliminary aggregated data based on the correspondence between buffer id and buffer;

[0102] The Memory server on the Mapper side is used to receive data pull requests from the Memory Client and pull the storage data corresponding to the buffer id from the Memory Storage on the Mapper side;

[0103] The Memory Client on the Reducer side is used to receive data pull requests from the ShuffleMemoryReader, transmit the pulled data, and record the buffer id and Memory server address of the pulled data.

[0104] Memory Storage on the Reducer side, used to store the pulled data; and

[0105] ShuffleMemoryReader on the Reducer side is used to read the preliminary aggregated data stored in the Memory Storage on the Reducer side and output the secondary aggregated data.

[0106] In a specific embodiment of the present invention, Figure 6 As shown in the figure, the process of non-invasively loading Shuffle in Spark application includes: comparing the ID of Spark application input by the user with the ID of native data shuffle transmission module. If the two are the same, Spark application loads the native data shuffle transmission module; if the two are not the same, Spark application non-invasively loads the user-defined module (i.e., memory data shuffle transmission module). In addition, if there is no data shuffle transmission module ID that is the same as the ID input by the user in Spark application, the system reports an error.

[0107] The data reading and writing method of the data shuffling transmission process of the Spark application mentioned in the present invention has the following advantages:

[0108] (1) Compared with the data writing module of the native Spark application, the memory writing module in the present invention can store the intermediate data directly in the memory, avoiding the performance bottleneck of the disk I / O, and significantly improving the execution speed of the entire Shuffle process (the reading speed is increased from 0.6 GB / s of the disk to 20 GB / s of the memory), thereby improving the efficiency of data processing.

[0109] (2) A non-intrusive improvement approach is adopted to design new components that can be loaded from outside the Spark application, including storage modules, transmission modules, and management modules. These components can be seamlessly integrated into the running environment of the Spark application through the native interface of the Spark application. There is no need to modify the source code and user code of the existing Spark application, and the modules are completely transparent to the existing Spark application.

[0110] Corresponding to the above method, the present invention also provides a data reading and writing system for the data shuffling transmission process of Spark application, the system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the system implements the steps of the method as described above.

[0111] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a magnetic disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0112] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0113] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0114] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data reading and writing method for a data shuffling transmission process of a Spark application, wherein the data shuffling transmission process includes a writing phase and a reading phase, and is characterized in that: The method comprises the following steps: In the writing phase, the memory writing module acquires data, converts it into preliminary aggregated data, and sends the preliminary aggregated data in batches to the memory storage module corresponding to the writing phase; wherein the memory storage module corresponding to the writing phase includes a directory unit and a memory pool unit; The memory pool unit allocates memory blocks to store the preliminary aggregated data of each batch sent by the memory writing module, the directory unit generates a memory block ID, records the correspondence between the memory block ID and the memory block, and uploads the memory block ID of each batch of data to the memory writing module until all Spark tasks in the writing phase are executed; wherein the memory writing module includes a pre-stored memory server address, and each memory block ID of each batch of preliminary aggregated data has a corresponding memory server address; In the reading phase, the memory reading module obtains the memory block ID and the corresponding memory server address of each batch of data based on the memory writing module, and sends the data pull request, the obtained memory block ID and the obtained memory server address to the reading memory client; The memory client establishes a communication connection with the memory server corresponding to the write phase based on the data pull request, memory block ID and memory server address issued by the memory read module to pull the preliminary aggregated data stored in the memory storage module corresponding to the write phase, and the memory client stores the pulled preliminary aggregated data in the memory storage module corresponding to the read phase; The memory reading module reads the preliminary aggregated data stored in the memory storage module corresponding to the reading phase, and outputs the secondary aggregated data until all Spark tasks in the reading phase are executed.

2. The method according to claim 1, characterized in that: Before sending the preliminary aggregated data in batches to the memory storage module corresponding to the writing stage, it also includes: determining the storage capacity of the memory storage module corresponding to the writing stage, and when the storage capacity is less than a set threshold, writing the preliminary aggregated data sent by the memory writing module to the disk.

3. The method according to claim 1, characterized in that After the reading phase, the method further includes: deleting the preliminary aggregated data stored in the memory storage module corresponding to the writing phase; The preliminary aggregated data stored in the memory storage module corresponding to the write phase is deleted in the following way: Positioning step: The directory unit locates the corresponding memory block of the memory pool unit based on the correspondence between the recorded memory block ID and the memory block; Removal step: deleting the preliminary aggregated data stored in the memory block corresponding to the memory pool unit, and removing the corresponding relationship between the memory block ID and the memory block from the directory unit; The positioning step and the removing step are repeated until all preliminary aggregated data stored in the memory storage module corresponding to the written stage are deleted.

4. The method according to claim 1, characterized in that: The memory writing module obtains data and converts it into preliminary aggregated data, including: The memory writing module obtains raw data from the native Spark application through the first interface and converts the raw data into preliminary aggregated data; or The memory writing module obtains preliminary aggregated data from the native Spark application through the first interface; The memory reading module reads the preliminary aggregated data stored in the memory storage module corresponding to the reading phase, and outputs secondary aggregated data, including: The memory reading module reads the preliminary aggregated data stored in the memory storage module corresponding to the reading phase, and outputs the secondary aggregated data to the native Spark application through the second interface.

5. The method according to claim 1, characterized in that: The memory client establishes a communication connection with the memory server corresponding to the write phase based on the data pull request, memory block ID and memory server address issued by the memory read module to pull the preliminary aggregated data stored in the memory storage module corresponding to the write phase, including: The memory client receives the data pull request, memory block ID and memory server address issued by the memory read module, and sends the data pull request and memory block ID to the specified memory server based on the memory server address; The memory server receives the data pull request and locates the corresponding memory block in the memory storage module corresponding to the write phase based on the acquired memory block ID, so that the preliminary aggregated data stored in the corresponding memory block can be pulled by the corresponding memory client in the read phase.

6. The method according to claim 5, characterized in that The method of locating a corresponding memory block in a memory storage module corresponding to the writing phase based on the acquired memory block ID includes: The memory server sends the acquired memory block ID to the memory storage module corresponding to the write phase; The memory storage module corresponding to the writing phase locates the memory block corresponding to the memory pool unit through the correspondence between the memory block ID recorded in the directory unit and the memory block and the acquired memory block ID.

7. The method according to claim 1, characterized in that The memory server is created using the Netty framework.

8. The method according to claim 1, characterized in that Before the data pull request, the obtained memory block ID and the obtained memory server address are sent to the memory client for reading, it also includes: the memory reading module determines whether the memory server address is consistent with the local address in the memory client; if the memory server address is consistent with the local address, the memory reading module reads data from the memory storage module corresponding to the reading phase; if the memory server address is inconsistent with the local address, the memory reading module sends a data pull request to the memory client.

9. A data reading and writing system for a data shuffling and transmission process of a Spark application, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images