A method and system for limiting memory used by Spark task shards

Abstract

The present application discloses a method and system for limiting the memory used by Spark task slices, the method comprising: a yarn process judgment step: starting the Spark task slice to scan each node of the Spark task slice at a fixed time and judging whether there is a Spark subprocess; a recording step: if the Spark subprocess exists, recording the task ID of the Spark task that starts the Spark subprocess; a classification step: traversing all yarn task processes on the node, and classifying the yarn task process and the Spark subprocess according to the task ID; a processing step: controlling the number of the yarn task process and the Spark subprocess in each category according to a preset value. The present invention solves the problem of excessive node memory usage and node downtime due to the Spark task starting multiple subprocesses.

Description

Technical Field

[0001] The present invention belongs to the field of methods for limiting memory used by Spark task slices, and in particular relates to a method and system for limiting memory used by Spark task slices. Background Art

[0002] When a Spark task is executed on YARN, an Executor process is started on the machine node. The memory used by the Executor process is managed by YARN. YARN will limit the number of Executor processes to control the overall memory usage. However, the Executor processes started by some Spark tasks will start child processes on the node (the most common one is the pyspark process). These child processes are not managed by YARN, which will cause the machine to crash due to excessive memory usage, affecting the normal execution of other tasks. Summary of the invention

[0003] The embodiments of the present application provide a method and system for limiting the memory used by Spark task shards, so as to at least solve the problem of subjective factors in related technologies.

[0004] The present invention provides a method for limiting the memory used by Spark task shards, which includes:

[0005] YARN process judgment step: start the Spark task slice to scan each node of the Spark task slice at a fixed time and judge whether there is a Spark subprocess;

[0006] Recording step: if the Spark subprocess exists, recording the task ID of the Spark task that starts the Spark subprocess;

[0007] Classification step: traverse all YARN task processes on the node, and classify the YARN task process and the Spark subprocess according to the task ID;

[0008] Processing steps: Control the number of the yarn task processes and the Spark subprocesses in each category according to preset values.

[0009] In the above method for limiting the memory used by Spark task shards, the classification step includes:

[0010] Among them, the Spark subprocesses that meet the task ID are classified into a class of processes and the limit value of the Spark subprocesses of this class is set to 2, and the yarn processes that do not meet the task ID are classified into a class of processes and the limit value of the yarn processes of this class is set to 10.

[0011] In the above method for limiting the memory used by Spark task shards, the scanning and judging step includes:

[0012] A scan is performed every 30 seconds to determine whether the Spark subprocess appears.

[0013] In the above method for limiting the memory used by Spark task shards, the processing steps include:

[0014] When the number of the two types of processes has reached the limit value, the redundant processes are deleted according to the limit value, and the node no longer receives new task processes;

[0015] When the number of both types of processes has not reached the limit value, a new task process can be assigned to the node. After judgment is made based on the task ID of the new task process, the new task process can be assigned to one of the two types of processes based on the judgment result, and then it is judged whether the process of this type reaches the limit value.

[0016] The above method for limiting the memory used by Spark task shards also includes:

[0017] Steps for resuming normal work: when it is scanned that the Spark subprocess on the node has been executed and no new Spark subprocess appears, the node is restored to a normal node and no longer intercepts any task process.

[0018] The present invention also includes a system for limiting the memory used by Spark task shards, characterized by comprising:

[0019] YARN process judgment module: starts the Spark task slice to scan each node of the Spark task slice at a fixed time and judges whether there is a Spark subprocess;

[0020] A recording module, wherein if the Spark subprocess exists, the recording module records the task ID of the Spark task that starts the Spark subprocess;

[0021] A classification module, traversing all YARN task processes on the node, and classifying the YARN task process and the Spark subprocess according to the task ID;

[0022] A processing module controls the number of the yarn task processes and the Spark subprocesses in each category according to preset values.

[0023] The above-mentioned system for limiting the memory used by Spark task shards, wherein the classification module classifies the Spark subprocesses that match the task ID into a class of processes and sets the limit value of the Spark subprocesses of this class to 2, and classifies the yarn processes that do not match the task ID into a class of processes and sets the limit value of the yarn processes of this class to 10.

[0024] In the above-mentioned system for limiting the memory used by Spark task shards, the scanning and judging module performs a scan every 30 seconds to judge whether the Spark subprocess appears.

[0025] The above-mentioned system for limiting the memory used by Spark task shards, wherein the processing module includes:

[0026] When the number of the two types of processes has reached the limit value, the redundant processes are deleted according to the limit value, and the node no longer receives new task processes;

[0027] When the number of both types of processes has not reached the limit value, a new task process can be assigned to the node. After judgment is made based on the task ID of the new task process, the new task process can be assigned to one of the two types of processes based on the judgment result, and then it is judged whether the process of this type reaches the limit value.

[0028] The above-mentioned system for limiting the memory used by Spark task shards, among others, includes:

[0029] The normal working recovery module, when it is scanned that the Spark subprocess on the node has been executed and no new Spark subprocess appears, the normal working recovery module restores the node to a normal node and no longer intercepts any task process.

[0030] The beneficial effects of the present invention are:

[0031] The present invention solves the problem of excessive node memory usage and node downtime caused by starting multiple sub-processes due to Spark tasks; different processes are classified on the node to ensure the normal execution of the sub-processes and other normal YARN tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0033] In the attached picture:

[0034] Figure 1This is a flowchart of the method for limiting the memory used by Spark task shards;

[0035] Figure 2 This is a flowchart of the specific steps for limiting the memory used by Spark task shards;

[0036] Figure 3 It is a structural schematic diagram of a system for limiting the memory used by Spark task shards of the present invention;

[0037] Figure 4 is a framework diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0039] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0040] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0041] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0042] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0043] Before elaborating on each embodiment of the present invention in detail, the core inventive concept of the present invention is summarized and elaborated in detail through the following several embodiments.

[0044] Please refer to Figure 1 , Figure 1 This is a flowchart of the method for limiting the memory used by Spark task shards. Figure 1 As shown, the method for limiting the memory used by Spark task slices of the present invention includes:

[0045] Scanning judgment step S1: Start the Spark task slice to scan the Spark at a fixed time interval

[0046] Each node of the task shard is scanned to determine whether a Spark subprocess appears;

[0047] Recording step S2: if the Spark subprocess exists, recording the task ID of the Spark task that starts the Spark subprocess;

[0048] Classification step S3: traversing all YARN task processes on the node, and classifying the YARN task process and the Spark subprocess according to the task ID;

[0049] Processing step S4: Control the number of the YARN task processes and the Spark subprocesses in each category according to preset values.

[0050] The classification step S3 comprises:

[0051] Among them, the Spark subprocesses that meet the task ID are classified into a class of processes and the limit value of the Spark subprocesses of this class is set to 2, and the yarn processes that do not meet the task ID are classified into a class of processes and the limit value of the yarn processes of this class is set to 10.

[0052] In the above method for limiting the memory used by Spark task shards, the scanning judgment step S1 includes:

[0053] A scan is performed every 30 seconds to determine whether the Spark subprocess appears.

[0054] In the above method for limiting the memory used by Spark task shards, the processing step S4 includes:

[0055] When the number of the two types of processes has reached the limit value, the redundant processes are deleted according to the limit value, and the node no longer receives new task processes;

[0056] When the number of both types of processes has not reached the limit value, a new task process can be assigned to the node. After judgment is made based on the task ID of the new task process, the new task process can be assigned to one of the two types of processes based on the judgment result, and then it is judged whether the process of this type reaches the limit value.

[0057] The above method for limiting the memory used by Spark task shards also includes:

[0058] Resuming normal working step S5: When it is scanned that the Spark subprocess on the node has been executed and no new Spark subprocess appears, the node is restored to a normal node and no longer intercepts any task process.

[0059] The following examples are given to specifically illustrate the method of limiting the memory used by Spark task slices of the present invention.

[0060] Embodiment 1:

[0061] like Figure 2 The specific steps of the method for limiting the memory used by Spark task shards of the present invention are as follows:

[0062] Install this device on each NodeManager node;

[0063] After the device is started, it will become a background daemon process, scanning every 30 seconds to determine whether there is a Spark subprocess;

[0064] When the Spark subprocess is found, the subprocess management process starts working;

[0065] Record the task ID of the Spark task that starts the child process. The number of processes with this task ID on the node cannot exceed 2.

[0066] Traverse all YARN task processes on this node and divide the processes into two categories according to the task ID in the previous step;

[0067] Spark processes that match the task ID are classified into one category, and the number of processes is managed. If the number of processes exceeds 2, the executing task process is randomly selected and killed until the number of processes drops to 2.

[0068] The yarn processes that do not match the task ID are classified into one category. The number of processes in this category is limited to 10. If the number of processes is greater than 10, processes are randomly selected and killed until the number of processes drops to 10.

[0069] The sub-process management process will store the quantity information of the two types of processes and synchronize the process quantity information in real time;

[0070] When the number of both types of processes has reached the limit, this node will no longer accept new task processes;

[0071] When the number of processes does not reach the limit, YARN can assign a new task process to this node. The sub-process management process will intercept the new task process and determine which category the new process belongs to based on its task ID. It will also determine whether the process reaches the limit. If it does not reach the limit, it will be executed normally. If it reaches the limit, the task process will be rejected.

[0072] If the subprocess management process scans and finds that the Spark subprocess on the node has been executed and no new Spark subprocess appears, the node will be restored to a normal node and no longer intercept any task process.

[0073] Based on the above method, the present invention manages the subprocess started by the Spark task in the form of a background daemon process, and will not affect the node when there is no subprocess; at the same time, it solves the problem of machine crash caused by excessive memory usage due to the inability of yarn to manage Spark subprocesses; and the limit value of the number of processes can be freely adjusted according to the machine configuration.

[0074] Embodiment 2:

[0075] Please refer to Figure 3 , Figure 3 Schematic diagram of the structure of the system for limiting the memory used by Spark task slices of the present invention. Figure 3 A system for limiting the memory used by Spark task shards according to the present invention is shown, which includes:

[0076] Scanning and judging module: Start the Spark task slice to scan each node of the Spark task slice at a fixed time and judge whether there is a Spark subprocess;

[0077] A recording module, wherein if the Spark subprocess exists, the recording module records the task ID of the Spark task that starts the Spark subprocess;

[0078] A classification module, traversing all YARN task processes on the node, and classifying the YARN task process and the Spark subprocess according to the task ID;

[0079] A processing module controls the number of the yarn task processes and the Spark subprocesses in each category according to preset values.

[0080] The above-mentioned system for limiting the memory used by Spark task shards, wherein the classification module classifies the Spark subprocesses that match the task ID into a class of processes and sets the limit value of the Spark subprocesses of this class to 2, and classifies the yarn processes that do not match the task ID into a class of processes and sets the limit value of the yarn processes of this class to 10.

[0081] In the above-mentioned system for limiting the memory used by Spark task shards, the scanning and judging module performs a scan every 30 seconds to judge whether the Spark subprocess appears.

[0082] The above-mentioned system for limiting the memory used by Spark task shards, wherein the processing module includes:

[0083] When the number of the two types of processes has reached the limit value, the redundant processes are deleted according to the limit value, and the node no longer receives new task processes;

[0084] When the number of both types of processes has not reached the limit value, a new task process can be assigned to the node. After judgment is made based on the task ID of the new task process, the new task process can be assigned to one of the two types of processes based on the judgment result, and then it is judged whether the process of this type reaches the limit value.

[0085] The above-mentioned system for limiting the memory used by Spark task shards, among others, includes:

[0086] The normal working recovery module, when it is scanned that the Spark subprocess on the node has been executed and no new Spark subprocess appears, the normal working recovery module restores the node to a normal node and no longer intercepts any task process.

[0087] Embodiment three:

[0088] Combination Figure 4 As shown, this embodiment discloses a specific implementation of a computer device. The computer device may include a processor 81 and a memory 82 storing computer program instructions.

[0089] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0090] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0091] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0092] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any one of the methods for limiting the memory used by the Spark task slices in the above embodiments.

[0093] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 4 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0094] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0095] The bus 80 includes hardware, software or both, and couples the components of the computer device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infini Band interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0096] The computing device can be combined based on the memory limit used by the Spark task slices. Figure 1-Figure 2 Describe the method.

[0097] In addition, in combination with the method for limiting the memory used by Spark task shards in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the methods for limiting the memory used by Spark task shards in the above embodiment is implemented.

[0098] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] In summary, based on the beneficial effects of the present invention, this patent provides a method for limiting the memory used by Spark task shards, which solves the problem of excessive node memory usage and node crashes caused by Spark tasks starting multiple sub-processes; different processes are classified on the node to ensure the normal execution of sub-processes and other normal yarn tasks.

[0100] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for limiting the memory used by Spark task shards, characterized in that: include: YARN process judgment step: start the Spark task slice to scan each node of the Spark task slice at a fixed time and judge whether there is a Spark subprocess; Recording step: if the Spark subprocess exists, recording the task ID of the Spark task that starts the Spark subprocess; Classification step: traverse all YARN task processes on the node, and classify the YARN task process and the Spark subprocess according to the task ID; Processing steps: controlling the number of the yarn task processes and the Spark subprocesses in each category according to preset values; The classification step include: The Spark subprocesses that match the task ID are classified into a class of processes and the limit value of the Spark subprocesses of this class is set to 2; the yarn processes that do not match the task ID are classified into a class of processes and the limit value of the yarn processes of this class is set to 10.

2. The method for limiting the memory used by Spark task shards as claimed in claim 1, characterized in that: The yarn process judgment step includes: A scan is performed every 30 seconds to determine whether the Spark subprocess appears.

3. The method for limiting the memory used by Spark task shards as claimed in claim 1, characterized in that: The processing steps include: When the number of both types of processes has reached the limit value, redundant processes are deleted according to the limit value, and the node no longer receives new task processes; When the number of both types of processes has not reached the limit value, a new task process can be assigned to the node. After judgment is made based on the task ID of the new task process, the new task process can be assigned to one of the two types of processes based on the judgment result, and then it is judged whether the process of this type reaches the limit value.

4. The method for limiting the memory used by Spark task shards as claimed in claim 3, characterized in that: Also includes: Steps for resuming normal work: when it is scanned that the Spark subprocess on the node has been executed and no new Spark subprocess appears, the node is restored to a normal node and no longer intercepts any task process.

5. A system for limiting the memory used by Spark task shards, characterized in that: include: YARN process judgment module: starts the Spark task slice to scan each node of the Spark task slice at a fixed time and judges whether there is a Spark subprocess; A recording module, wherein if the Spark subprocess exists, the recording module records the task ID of the Spark task that starts the Spark subprocess; A classification module, traversing all YARN task processes on the node, and classifying the YARN task process and the Spark subprocess according to the task ID; A processing module, wherein the processing module controls the number of the yarn task processes and the Spark subprocesses in each category according to preset values; Among them, the classification module classifies the Spark subprocesses that meet the task ID into a class of processes and sets the limit value of the Spark subprocesses of this class to 2, and classifies the yarn processes that do not meet the task ID into a class of processes and sets the limit value of the yarn processes of this class to 10.

6. The system for limiting the memory used by Spark task shards as claimed in claim 5, characterized in that: The yarn process determination module performs a scan every 30 seconds to determine whether the Spark subprocess appears.

7. The system for limiting the memory used by Spark task shards as claimed in claim 5, characterized in that: The processing module comprises: When the number of both types of processes has reached the limit value, redundant processes are deleted according to the limit value, and the node no longer receives new task processes; When the number of both types of processes has not reached the limit value, a new task process can be assigned to the node. After judgment is made based on the task ID of the new task process, the new task process can be assigned to one of the two types of processes based on the judgment result, and then it is judged whether the process of this type reaches the limit value.

8. The system for limiting the memory used by Spark task shards as claimed in claim 7, characterized in that: Also includes: The normal working recovery module, when it is scanned that the Spark subprocess on the node has been executed and no new Spark subprocess appears, the normal working recovery module restores the node to a normal node and no longer intercepts any task process.

Images