MapReduce memory management method and system
By introducing the memory manager module, dynamically adjusting memory allocation, the problem of unreasonable memory configuration in MapReduce jobs is solved, resource utilization and task execution efficiency are improved, disk read and write burden is reduced, and system stability is ensured.
Patent Information
- Application Number
- CN202510317094.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-18
- Publication Date
- 2025-07-11
AI Technical Summary
The existing MapReduce job memory management strategy lacks dynamic and intelligentness, resulting in unreasonable memory configuration, affecting task execution efficiency and system stability, and the problem is more significant especially when dealing with large-scale data.
The memory manager module is introduced to monitor memory usage in real time, dynamically adjust the size of each cache area, allocate memory resources reasonably according to task requirements, and manage multiple memory users uniformly through the memory manager module to avoid resource waste and performance degradation.
It improves memory resource utilization, optimizes overwrite operations, reduces the burden of disk read and write tasks, ensures task execution efficiency and system stability, and is especially suitable for high I/O-intensive task scenarios.
Smart Images
Figure CN120295764A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of memory management, and more particularly to a memory management method and system for MapReduce. Background Art
[0002] Memory management and optimization of MapReduce refer to the management and optimization of the memory used in processes such as data processing, partitioning, aggregation, and sorting in MapTask and ReduceTask. For example, currently MapReduce strictly divides and controls the memory usage during the data spill process of MapTask. Specifically, when the in-memory data in the MapTask circular buffer reaches a certain threshold, MapTask will spill the data to disk for temporary storage to leave more memory for storing subsequent data. An effective optimization measure for MapReduce jobs is to customize and modify these configuration parameters according to the data characteristics of various applications to ensure minimizing the number of interactions (data) between MapTask and disk as much as possible.
[0003] Existing memory management and optimization of MapReduce jobs often require experienced engineers to manually adjust parameters according to specific application scenarios, data characteristics, and task types. For example, the setting of key parameters such as the memory allocation ratio, spill threshold, and buffer size usually depends on an in-depth understanding of the data scale, distribution, and operation mode. For inexperienced users or when facing complex and changeable data characteristics, this method may lead to unreasonable memory configuration, ultimately affecting the task execution efficiency and system stability.
[0004] The current memory management strategy lacks dynamics and intelligence in design and often adopts fixed or static allocation schemes. For example, memory may be allocated to operation modules such as data processing, sorting, and partitioning according to a preset ratio. However, during the actual execution of tasks, the memory requirements of each module may change significantly. Due to the lack of a flexible adjustment mechanism, some modules may experience insufficient memory, resulting in performance degradation, program errors, etc., while some other modules may relatively waste memory, thus failing to fully utilize the hardware resource advantages of the system. Especially when dealing with large-scale data, the problems caused by improper resource allocation will become more prominent.
[0005] To establish an AI-related model that can ensure correctness and stability, it is first necessary to execute various types of MapReduce jobs a certain number of times. This process may consume more computing resources. At the same time, although prior calculations provide the necessary data basis for model training, these data may not be sufficient to cover all potential job scenarios, resulting in performance degradation or maladaptation of the model when facing certain specific situations. In addition, since the optimization strategy is dynamically generated based on specific types of prior data, when the task environment changes, the effectiveness of these strategies may be affected, thereby limiting the overall generality and stability of the model.
[0006] That is, in the current industry, such optimization work mainly relies on enterprises to recruit professional high-end talents or build an optimization model through AI-related means to complete the optimization work. The common feature of these methods is to rely on a designed fixed parameter table: the optimization algorithm usually needs to preset the parameter table or training data set based on clear goals, rules, and constraints, and execute a series of optimization projects. Although this fixed mode can ensure the stability of the optimization process, in the face of a dynamically changing environment or complex factors, how to more efficiently adjust the parameter table or introduce a dynamic optimization mechanism has become an important research direction in the optimization field. Summary of the Invention
[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a memory management method and system for MapReduce, which can monitor the memory usage in real time in the face of a dynamically changing task environment or complex factors, and reasonably allocate memory resources according to task requirements.
[0008] To achieve the above purpose, the present invention is implemented by the following technical solutions: On the one hand, the present invention provides a memory management method for MapReduce, in which a memory manager module is added before the end of the Reduce stage in the MapReduce framework. The method includes: In response to the memory usage in the MapTask buffer reaching the task memory threshold and the memory manager module having additional memory, allocate the additional memory to the MapTask buffer, process the task data to obtain processed data, and store the processed data in the MapTask buffer; In response to the memory usage in the MapTask buffer reaching the task memory threshold, the memory manager module having no additional memory, and other memory users of the same task being able to release memory, allocate the memory released by other memory users of the same task to the MapTask buffer, process the task data to obtain processed data, and store the processed data in the MapTask buffer; In response to the memory usage of the MapTask buffer reaching the task memory threshold, the memory manager module having no additional memory, and other memory consumers of the same task being unable to release memory, the MapTask buffer triggers an overflow write mechanism and writes the processed data to the MapTask disk; In response to the memory usage of the ReduceTask buffer reaching a preset threshold and the memory manager module having additional memory, the additional memory is allocated to the ReduceTask buffer, the processed data is merged and sorted to obtain sorted data, and the sorted data is stored in the ReduceTask buffer; In response to the memory usage of the ReduceTask buffer reaching a preset threshold and the memory manager module having no additional memory, the ReduceTask buffer triggers an overflow write mechanism and writes the sorted data and the processed data to the ReduceTask disk.
[0009] Optionally, the task memory threshold is 80% of the MapTask buffer memory.
[0010] Optionally, before all the processed data enters the ReduceTask buffer, it further includes: Performing a final merge operation and a combine operation on the processed data to obtain operation data.
[0011] Optionally, it further includes: Reading the sorted data in the ReduceTask buffer and the ReduceTask disk to obtain the key-value pairs of the sorted data; Storing the key-value pairs of the sorted data in the additional memory of the memory manager module, and writing the key-value pairs of the sorted data still stored in the ReduceTask buffer to the ReduceTask disk.
[0012] Optionally, writing the key-value pairs of the sorted data still stored in the ReduceTask buffer to the ReduceTask disk includes: Writing the key-value pairs of the part of the sorted data with a later sort order still stored in the ReduceTask buffer to the ReduceTask disk.
[0013] Optionally, the initial memory of the MapTask buffer and the ReduceTask buffer is set according to task requirements.
[0014] In a second aspect, the present invention provides a memory management system for MapReduce, including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the method according to the first aspect.
[0015] In a third aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the method according to the first aspect are implemented.
[0016] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: 1. By dynamically adjusting the size of each buffer, the present invention avoids memory waste or shortage caused by fixation, thereby improving the utilization rate of memory resources, optimizing the triggering conditions of the spill write operation, and only performing spill write when it is inconvenient to expand the memory. This reduces the burden of disk read and write tasks, reduces disk I / O overhead, and through the dynamic adjustment of the memory manager module under changing task requirements, ensures task execution efficiency and system stability. The optimization of the spill write mechanism reduces the disk read and write load and is particularly suitable for use in high I / O-intensive task scenarios.
[0017] 2. The present invention adds a memory manager module to uniformly manage the memory resources of tasks; the memory manager module can monitor the memory usage in real time and dynamically allocate or reclaim memory according to task requirements. One memory manager module manages multiple other memory users of the same task (such as data processing, partitioning module, sorting module, etc.). The memory manager module can dynamically adjust the memory allocation ratio according to the memory requirements of each memory user during the task execution process, avoiding resource waste and performance degradation.
[0018] 3. When a certain memory user runs out of memory, the memory manager module can notify other memory users within the same task to release some memory. The memory manager module restores the released memory resources through the memory release mechanism and efficiently allocates them to the memory-deficient module, ensuring that the task can be executed under limited memory conditions. When the task memory is tight, by means of the release priority issue, only the spill write operation mechanism is triggered when the release is ineffective, thereby reducing the spill write frequency and the disk I / O burden. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 The figure shows a schematic flow chart of the memory management method for the MapTask buffer of the present invention in an embodiment; Figure 2 The figure shows a schematic flow chart of the memory management method for the ReduceTask buffer of the present invention in an embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0020] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. Without conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0021] The term "and / or" is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " generally represents an "or" relationship between the front and rear associated objects.
[0022] Embodiment 1
[0023] This embodiment introduces a memory management method for MapReduce. By introducing a memory manager module (MemoryManager) before the end of the Reduce stage in the programming model (MapReduce) framework, it is responsible for the dynamic allocation and refined management of memory. MemoryManager monitors the memory usage in real time and reasonably allocates memory resources according to task requirements. MemoryManager can dynamically manage the memory ratios of multiple memory consumers (MemoryConsumer) within the same task, such as data processing, partitioning, sorting, map functions, etc., to avoid resource waste and ensure that critical tasks can obtain sufficient memory support. In addition, MemoryManager provides a unified interface for memory application and release, simplifying the complexity of memory management. MemoryManager mainly processes the memory management in the MapTask stage and the memory management before the Reduce function in the ReduceTask stage. The memory management of MemoryManager is not involved during the processing of the Reduce function.
[0024] The method includes the following steps: Step 1: In response to the memory usage in the MapTask buffer reaching the task memory threshold and there being additional memory in the memory manager module, allocate the additional memory to the MapTask buffer, process the task data to obtain processed data, and store the processed data in the MapTask buffer; Step 2: In response to the memory usage in the MapTask buffer reaching the task memory threshold, there being no additional memory in the memory manager module, and other memory consumers of the same task being able to release memory, allocate the memory released by other memory consumers of the same task to the MapTask buffer, process the task data to obtain processed data, and store the processed data in the MapTask buffer; Step 3: In response to the memory usage of the MapTask buffer reaching the task memory threshold, the memory manager module having no extra memory, and other memory consumers of the same task being unable to release memory, the MapTask buffer triggers the spill - write mechanism to write the processed data to the MapTask disk; Step 4: In response to the memory usage of the ReduceTask buffer reaching the preset threshold and the memory manager module having extra memory, allocate the extra memory to the ReduceTask buffer, perform merge - sorting on the processed data, and store the sorted data in the ReduceTask buffer; Step 5: In response to the memory usage of the ReduceTask buffer reaching the preset threshold and the memory manager module having no extra memory, the ReduceTask buffer triggers the spill - write mechanism to write the sorted data and the processed data to the ReduceTask disk.
[0025] In this embodiment, a memory manager module is added to uniformly manage the memory resources of the task; the memory manager module can monitor the memory usage in real - time and dynamically allocate or reclaim memory according to the task requirements. One memory manager module manages multiple other memory consumers of the same task (such as data processing, partitioning module, sorting module, etc.). The memory manager module can dynamically adjust the memory allocation ratio according to the memory requirements of each memory consumer during the task execution process, avoiding resource waste and performance degradation.
[0026] Embodiment 2
[0027] Based on Embodiment 1, as Figure 1 、 2 shown, this embodiment introduces a memory management method for MapReduce, which specifically includes the following steps: For the MapTask stage, initialize a circular buffer, i.e., the MapTask buffer, at the start of MapTask. The initial memory size of the MapTask buffer is dynamically set according to the initial requirements of the task and the available memory of the system, avoiding frequent triggering of spill - write operations due to too small a memory size of the MapTask buffer or memory waste due to too large a memory size of the MapTask buffer. The design of the MapTask buffer can achieve efficient memory management and support fast writing and reading of data.
[0028] When the memory usage of the MapTask buffer exceeds the task memory threshold, it will apply to the MemoryManager for more memory: if the MemoryManager determines that there is still remaining memory resource available in the system, it will allocate additional memory to the MapTask buffer to meet the task requirements; if the MemoryManager does not have enough memory, the MemoryConsumer will notify other MemoryConsumers of the same task to release memory so that the MapTask buffer can apply for more content; if no other MemoryConsumer releases memory, the allocation of additional memory will stop, triggering the spilling operation of the data in the MapTask buffer.
[0029] When the newly added memory resource reaches the threshold again, the above application process can be repeated. Through this mechanism, the system realizes the dynamic expansion and efficient utilization of memory, avoiding the problems of resource waste or insufficiency caused by the fixed allocation strategy.
[0030] In a specific embodiment, to make the content of MapTask clearer and easier to understand, as Figure 1 shown, it includes: Step 1: In the initialization stage of MapTask, the total memory resource allocated by the system for a single MapTask is 3GB, the initial memory size of the MapTask buffer is set to 1GB, and the task memory threshold is 80% of the MapTask buffer memory, that is, 800MB. This MapTask buffer is used to store the processed data, including the data generated during partitioning, sorting, and aggregation.
[0031] Step 2: When the memory usage of the MapTask buffer reaches 800MB, MapTask initiates a memory expansion application to the MemoryManager. The MemoryManager checks the global memory usage status of the system and confirms that there is still 1.5GB of available memory resource. Therefore, it allocates an additional 500MB of memory to the MapTask buffer, expanding the total memory size of the MapTask buffer to 1.5GB.
[0032] Step 3: The data processing continues to obtain processed data, which is stored in the MapTask buffer. When the memory usage of the MapTask buffer reaches 1.2 GB, the task memory threshold is 80% of the total memory size of the extended MapTask buffer at this time. The MapTask applies to the MemoryManager for more memory again. If the available memory of the system is insufficient at this time, the MemoryManager will notify other MemoryConsumers of the same task to release memory so that the MapTask buffer can apply for more content. If other MemoryConsumers release a certain amount of memory, the MemoryManager will transfer this part of the memory to the MapTask buffer. If no other MemoryConsumers release memory, the MemoryManager stops allocating additional memory. At this time, the buffer triggers the spill-write mechanism to write the processed data into the MapTask disk temporary file, and the unprocessed task data is temporarily persisted.
[0033] Step 4: When subsequent data flows into the MapTask buffer, whenever the usage of the MapTask buffer approaches the task memory threshold, the MapTask first tries to apply to the MemoryManager for more memory resources. If the memory is insufficient, it preferentially releases the memory of other MemoryConsumers of the same task. When the memory cannot be released, the processed data is partially written to the MapTask disk through the spill-write mechanism, and the unprocessed task data is temporarily persisted.
[0034] Through the combination of dynamic adjustment and spill-write in this process, it is ensured that the task can run efficiently under limited memory conditions.
[0035] Step 5: After all task data is processed in the MapTask buffer, finalmerge operation, combine operation, etc. are performed on the processed data to obtain the operation data, ensuring that all processed data is available for the reduction task (ReduceTask).
[0036] Similarly, for the ReduceTask stage, the ReduceTask buffer is initialized at the start of the ReduceTask. The ReduceTask buffer is used to store the data output by the MapTask pulled during the shuffle process, that is, the operation data. Memory is continuously applied to the MemoryManager during the processes of data pulling, merging, and sorting to ensure that the data does not overflow to the disk as much as possible.
[0037] In a specific embodiment, to make the content of the ReduceTask clearer and easier to understand, as Figure 2 shown, it includes: Step 1: During the initialization phase of the ReduceTask, the total memory resource allocated by the system for a single ReduceTask is 3GB, and the initial memory size of the ReduceTask buffer is set to 1GB.
[0038] Step 2: During the shuffle process, the ReduceTask merges and sorts the pulled operation data, sorts it by key value, and stores the sorted data in the ReduceTask buffer; when the memory usage of the ReduceTask buffer approaches the preset threshold, the ReduceTask first attempts to apply to the MemoryManager for more memory resources to complete the merging and sorting operations; if the memory application fails, the ReduceTask triggers the spill mechanism, writing part of the sorted data to the temporary file on the ReduceTask disk, but the unsorted and unmerged operation data is also written to the temporary file on the ReduceTask disk.
[0039] In a specific embodiment, for the sorted data, if the ReduceTask buffer has enough memory, it is first stored in the ReduceTask buffer, and if the ReduceTask buffer has reached the preset threshold and the memory management module cannot allocate additional memory, it is stored on the ReduceTask disk; for the unsorted and unmerged data, i.e., the processed data, it is first stored in the ReduceTask buffer and spilled to the ReduceTask disk when the ReduceTask buffer is full and the memory management module cannot allocate additional memory.
[0040] Step 3: The ReduceTask continuously reads the sorted data from the ReduceTask buffer and the ReduceTask disk to obtain the key-value pairs of the sorted data; and applies to the MemoryManager for additional memory to store the key-value pairs of the sorted data as needed. At the same time, the MemoryManager notifies the sorted data still stored in the ReduceTask buffer to release some content, and the way to release it is to spill some of the sorted data at the end of the sort to the ReduceTask disk.
[0041] Step 4: After the Reduce Task completes the merge sorting, it calls the Reduce function to calculate the key-value pairs of the sorted data, calculates multiple values under the same key, such as summation, maximum value, counting, etc., to obtain the calculation result. The final calculation result is a Key and an aggregated value, and finally outputs the file and stores it in the distributed file system; Among them, the Reduce function releases the memory resources of a group of key-value pairs after calculating one group.
[0042] In this embodiment, by dynamically adjusting the size of each buffer, it is possible to avoid memory waste or insufficiency caused by fixation, thereby improving the utilization rate of memory resources, optimizing the triggering conditions of the spill-write operation, and only performing spill-write when memory expansion is inconvenient, reducing the burden of disk read / write tasks and the disk I / O overhead. Under variable task requirements, through the dynamic adjustment of the memory manager module, the task execution efficiency and system stability are ensured. The optimization of the spill-write mechanism reduces the disk read / write load and is particularly suitable for use in high I / O-intensive task scenarios.
[0043] Embodiment 3
[0044] This embodiment introduces a memory management system for MapReduce, including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the method described in Embodiment 1.
[0045] Embodiment 4
[0046] This embodiment introduces a computer-readable storage medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the method described in Embodiment 1 are implemented.
[0047] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0048] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more flows or multiple flows and / or blocks Figure 1 one or more blocks or multiple blocks.
[0049] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more of the flows Figure 1 and / or boxes of one or more of the flows Figure 1 and / or boxes.
[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the flows Figure 1 and / or boxes of one or more of the flows Figure 1 and / or boxes.
[0051] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit of the present invention and the scope protected by the claims. All of these are within the protection scope of the present invention.
Claims
1. A memory management method for MapReduce, characterized in that, Before the end of the Reduce phase in the MapReduce framework, adding a memory manager module, the method comprising: In response to the memory usage of the MapTask buffer reaching the task memory threshold and the memory manager module having additional memory, allocating the additional memory to the MapTask buffer, processing the task data to obtain processed data, and storing the processed data in the MapTask buffer; In response to the memory usage of the MapTask buffer reaching the task memory threshold, the memory manager module not having additional memory, and other memory consumers of the same task being able to release memory, allocating the memory released by other memory consumers of the same task to the MapTask buffer, processing the task data to obtain processed data, and storing the processed data in the MapTask buffer; In response to the memory usage of the MapTask buffer reaching the task memory threshold, the memory manager module not having additional memory, and other memory consumers of the same task not being able to release memory, the MapTask buffer triggers an overflow write mechanism to write the processed data to the MapTask disk; In response to the memory usage of the ReduceTask buffer reaching a preset threshold and the memory manager module having additional memory, allocating the additional memory to the ReduceTask buffer, performing merge sorting on the processed data to obtain sorted data, and storing the sorted data in the ReduceTask buffer; In response to the memory usage of the ReduceTask buffer reaching a preset threshold and the memory manager module not having additional memory, the ReduceTask buffer triggers an overflow write mechanism to write the sorted data and the processed data to the ReduceTask disk.
2. The memory management method of MapReduce according to claim 1, wherein The task memory threshold is 80% of the memory of the MapTask buffer.
3. The memory management method of MapReduce according to claim 1, characterized in that, Before all the processed data enters the ReduceTask buffer, it further includes: Performing a final merge operation and a combine operation on the processed data to obtain operation data.
4. The memory management method of MapReduce according to claim 1, wherein It further includes: Reading the key-value pairs of the sorted data in the ReduceTask buffer and the ReduceTask disk to obtain the key-value pairs of the sorted data; Storing the key-value pairs of the sorted data in the additional memory of the memory manager module, and writing the key-value pairs of the sorted data still stored in the ReduceTask buffer to the ReduceTask disk.
5. The memory management method of MapReduce according to claim 4, wherein Writing the key-value pairs of the sorted data still stored in the ReduceTask buffer to the ReduceTask disk, including: Writing the key-value pairs of the partially sorted data with a later sort order still stored in the ReduceTask buffer to the ReduceTask disk.
6. The memory management method of MapReduce according to claim 1, wherein The initial memory of the MapTask buffer and the ReduceTask buffer is set according to the task requirements.
7. A memory management system for MapReduce, characterized in that Including a processor and a storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the method according to any one of claims 1 to 7.
8. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the computer instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.