Physical memory overrun determination method and device for Flink job, equipment and storage medium
By determining the job identification of the Flink job and the physical memory usage of the process, we can determine whether the physical memory of the Flink ON YARN job exceeds the limit, and solve the problem of failure to effectively detect the physical memory exceeding the limit in the existing technology, and realize the reasonable allocation of resources and Efficient use improves the stability of the operation.
Patent Information
- Application Number
- CN202411580751.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-06
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art has failed to effectively detect whether the physical memory of the Flink ON YARN job exceeds the limit, resulting in resource waste and stability issues.
By determining the job identity of the Flink job, obtaining the physical memory usage of multiple corresponding processes, calculating the total physical memory used, and comparing it with the allocated physical memory to determine whether the limit is exceeded.
It realizes accurate monitoring of the physical memory usage of Flink jobs, avoids resource waste, ensures reasonable allocation and efficient utilization of resources, and improves the stability of jobs.
Smart Images
Figure CN119440848A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of job memory determination, and in particular to a method, apparatus, device and storage medium for determining physical memory limit excess of a Flink job. Background Art
[0002] In recent years, with the mature development of Flink jobs, a more practical stream processing application development and management framework, Stream Park, has emerged. This open source project makes up for the pain point of not having a deployment management platform for Flink and Spark jobs. Based on Stream Park, developers can easily build and manage stream processing applications and better use Apache Flink to write a development framework for stream processing applications.
[0003] However, despite the industry's many improvements in Flink job performance monitoring, the community's main focus is still on TaskManager's heap memory and other performance monitoring indicators, and the performance tuning of heap memory is limited to simple calculations and has not yet gone deep into the management of physical memory. In addition, the current monitoring and tuning measures do not involve the detection of memory limit violations.
[0004] In summary, the current work on detecting whether the physical memory usage of a Flink job running on a YARN cluster exceeds the limit is still imperfect. Therefore, the present invention provides a method for detecting whether the physical memory usage of a Flink ON YARN job exceeds the limit. Summary of the invention
[0005] The present application provides a method, apparatus, device and storage medium for determining physical memory overlimit of a Flink job, so as to solve the problem that the existing monitoring system cannot effectively detect physical memory overlimit.
[0006] In a first aspect, the present application provides a method for determining physical memory limit excess of a Flink job, the method comprising:
[0007] Determine the job identifier corresponding to the Flink job;
[0008] According to the job identifier, determine multiple processes corresponding to the Flink job, and determine physical memory used by the multiple processes, wherein the multiple processes include: multiple task processes and one job process;
[0009] Determine the total physical memory used by the Flink job according to the physical memory used by the multiple processes;
[0010] Determine the allocated physical memory of the Flink job, and determine whether the total physical memory of the Flink job exceeds a limit according to the allocated physical memory and the total used physical memory.
[0011] Optionally, determining the physical memory used by the multiple processes includes:
[0012] For any one of the multiple processes, determine process information of the process, the process information including: a process number;
[0013] Determine the system file corresponding to the process number, and determine the anonymous physical memory, file physical memory and shared memory corresponding to the system file;
[0014] The sum of the anonymous physical memory, the file physical memory and the shared memory is taken as the physical memory used by the process.
[0015] Optionally, determining the total physical memory used by the Flink job according to the physical memory used by the multiple processes includes:
[0016] The sum of the physical memories used by the multiple processes is taken as the total physical memory used by the Flink job.
[0017] Optionally, the process information further includes: process command information, and the method further includes:
[0018] Determine, according to the process command information, a character string corresponding to the process;
[0019] Segment the character string to obtain multiple arrays;
[0020] Using an array at a preset position among the multiple arrays as the process name of the process;
[0021] A first physical memory table corresponding to the Flink job is generated according to the process names and used physical memory corresponding to the multiple processes.
[0022] Optionally, the allocated physical memory includes: the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process, and the determining the allocated physical memory of the Flink job, and judging whether the total physical memory of the Flink job exceeds the limit according to the allocated physical memory and the total used physical memory, includes:
[0023] Perform data analysis and processing according to the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process to obtain the allocated physical memory of the Flink job;
[0024] The allocated physical memory is compared with the total physical memory to determine whether the total physical memory of the Flink job exceeds the limit.
[0025] Optionally, performing data analysis and processing according to the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process to obtain the allocated physical memory of the Flink job includes:
[0026] According to the job identifier, determine the maximum parallelism of the Flink job, and determine the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process in the Flink job, and the number of the task processes;
[0027] The allocated physical memory of the Flink job is obtained by performing calculation based on the maximum parallelism, the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process, and the number of the task processes.
[0028] Optionally, comparing the allocated physical memory with the total physical memory to determine whether the total physical memory of the Flink job exceeds a limit includes:
[0029] If the total used physical memory is greater than the allocated physical memory, the total physical memory of the Flink job exceeds the limit;
[0030] If the total used physical memory is less than or equal to the allocated physical memory, the total physical memory of the Flink job does not exceed the limit.
[0031] In a second aspect, the present application provides a device for determining a physical memory limit excess of a Flink job, the device comprising:
[0032] The determination module is used to determine the job identifier corresponding to the Flink job;
[0033] The determination module is further used to determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine physical memory used by the multiple processes, wherein the multiple processes include: multiple task processes and one job process;
[0034] The determination module is further configured to determine the total physical memory used by the Flink job according to the physical memory used by the multiple processes;
[0035] The determination module is further used to determine the allocated physical memory of the Flink job;
[0036] A judgment module is used to judge whether the total physical memory of the Flink job exceeds the limit according to the allocated physical memory and the total used physical memory.
[0037] Optionally, the device further comprises: a processing module;
[0038] The determination module is further used to determine, for any one of the multiple processes, process information of the process, the process information including: a process number; determining a system file corresponding to the process number, and determining anonymous physical memory usage, file physical memory usage, and shared memory corresponding to the system file;
[0039] The processing module is used to take the sum of the anonymously used physical memory, the file used physical memory and the shared memory as the used physical memory of the process.
[0040] Optionally, the processing module is further configured to use the sum of the used physical memories of the multiple processes as the total used physical memory of the Flink job.
[0041] Optionally, the determination module is further used to determine a character string corresponding to the process according to the process command information;
[0042] The processing module is further used to segment the string to obtain multiple arrays; use the array at a preset position in the multiple arrays as the process name of the process; and generate a first physical memory table corresponding to the Flink job according to the process names corresponding to the multiple processes and the used physical memory.
[0043] Optionally, the processing module is further used to perform data analysis and processing according to the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process to obtain the allocated physical memory of the Flink job;
[0044] The judgment module is further used to compare the allocated physical memory with the total physical memory to determine whether the total physical memory of the Flink job exceeds the limit.
[0045] Optionally, the determination module is further used to determine the maximum parallelism of the Flink job according to the job identifier, and determine the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process in the Flink job and the number of the task processes;
[0046] The processing module is further used to perform calculations based on the maximum degree of parallelism, the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process, and the number of task processes to obtain the allocated physical memory of the Flink job.
[0047] Optionally, the determination module is further used to determine that the total physical memory of the Flink job exceeds the limit when the total physical memory is greater than the allocated physical memory; and to determine that the total physical memory of the Flink job does not exceed the limit when the total physical memory is less than or equal to the allocated physical memory.
[0048] In a third aspect, the present application provides a device for determining a physical memory limit excess of a Flink job, including:
[0049] Memory;
[0050] processor;
[0051] Wherein, the memory stores computer-executable instructions;
[0052] The processor executes the computer-executable instructions stored in the memory to implement the method for determining physical memory limit excess of a Flink job as described in the first aspect and various possible implementations of the first aspect.
[0053] In a fourth aspect, the present application provides a computer storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method for determining physical memory excess of a Flink job as described in the first aspect and various possible implementations of the first aspect.
[0054] The present application provides a method for determining the physical memory overlimit of a Flink job, by determining the job identifier corresponding to the Flink job to be detected, obtaining multiple processes contained in each Flink job to be detected, thereby obtaining the used physical memory of each process, and then determining the total used physical memory of each Flink job to be detected by calculation, and finally comparing the total used physical memory of each Flink job to be detected with its corresponding allocated physical memory to determine whether the used physical memory of each Flink job to be detected exceeds the limit. Accurate monitoring of the physical memory used by Flink jobs is achieved, avoiding waste of resources, ensuring the reasonable allocation and efficient use of resources, and improving the stability of jobs. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0056] Figure 1 The process of determining the physical memory limit of a Flink job provided in this application Figure 1 ;
[0057] Figure 2 The process of determining the physical memory limit of a Flink job provided in this application Figure 2 ;
[0058] Figure 3 A schematic diagram of the structure of a device for determining physical memory limit excess of a Flink job provided in this application;
[0059] Figure 4 A schematic diagram of the structure of a device for determining physical memory limit excess of a Flink job provided in this application.
[0060] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0062] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.
[0063] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0064] First, the terms involved in this application are explained:
[0065] Apache Flink: It is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data flow engine used for stateful computing of unbounded and bounded data streams. Flink can run in all common cluster environments and can perform calculations at memory speed and any scale; Flink runtime consists of two types of processes: a job manager (JobManager) and one or more task managers (TaskManager).
[0066] Job Manager: Mainly responsible for scheduling tasks, coordinating checkpoint error recovery, etc. When the client submits the packaged job to the JobManager, the JobManager will assign the job to a TaskManager with resources based on the registered TaskManager resource information, and then start running the job. TaskManager obtains task information from the JobManager, and then uses slot resources to run the task.
[0067] Task Manager: It is a task that executes data flow. A task may have multiple subtasks by setting the degree of parallelism. Each TaskManager runs as an independent JVM process. It is mainly responsible for executing operators in independent threads. The number of operators that can be executed depends on the number of slots specified by each taskManager. Task slot is the smallest resource unit in Flink. If a taskManager has 2 slots, it will allocate 1 / 2 of the memory resources to each slot. Slots in the same taskManager will share network resources and heartbeat information.
[0068] In recent years, with the mature development of Flink jobs, a more practical stream processing application development and management framework, Stream Park, has emerged. This open source project makes up for the pain point of not having a deployment management platform for Flink and Spark jobs. Based on Stream Park, developers can easily build and manage stream processing applications and better use Apache Flink to write a development framework for stream processing applications.
[0069] However, despite the industry's many improvements in Flink job performance monitoring, the community's main focus is still on TaskManager's heap memory and other performance monitoring indicators, and the performance tuning of heap memory is limited to simple calculations and has not yet gone deep into the management of physical memory. In addition, the current monitoring and tuning measures do not involve the detection of memory limit violations.
[0070] In response to the above problems, this application proposes a method for determining physical memory limit excess of a Flink job.
[0071] This method determines the job identifier corresponding to the Flink job to be detected, obtains multiple processes contained in each Flink job to be detected, and thus obtains the physical memory used by each process, and then determines the total physical memory used by each Flink job to be detected by calculation, and finally compares the total physical memory used by each Flink job to be detected with its corresponding allocated physical memory to determine whether the physical memory used by each Flink job to be detected exceeds the limit. This method realizes accurate monitoring of the physical memory used by Flink jobs, avoids waste of resources, ensures the reasonable allocation and efficient use of resources, and improves the stability of jobs.
[0072] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0073] Figure 1 This is a process of a method for determining physical memory limit excess of a Flink job provided in an embodiment of the present application. Figure 1 .like Figure 1 As shown, the method for determining physical memory limit excess of a Flink job provided in this embodiment includes:
[0074] S101: Determine a job identifier corresponding to a Flink job.
[0075] The job ID refers to an applicationId that each Flink job has. It is the unique identifier of each Flink job and can help users and systems manage it in a Flink cluster.
[0076] The purpose of determining the applicationId corresponding to a Flink job is to manage a specific job instance in a Flink cluster, for example, to determine whether the physical memory used by the job exceeds the limit, so as to facilitate the start, stop, monitor, and fault recovery of the job. Through the applicationId, users can accurately track and control the life cycle of the job to ensure efficient management and stable operation of the job.
[0077] In a possible implementation, by connecting to the Stream Park database, all Flink jobs that need to detect whether the physical memory usage exceeds the limit can be read out, and the applicationId value and job name corresponding to each Flink job can be retrieved.
[0078] S102: Determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine physical memory used by the multiple processes.
[0079] The multiple processes include: multiple task processes and one job process.
[0080] It is understandable that a Flink job contains multiple processes, and the physical memory usage of each process is different. Here, the physical memory used by each process of each job is calculated separately. According to the job identifier of the job to be detected, the multiple processes corresponding to the Flink job can be determined, and thus the physical memory used by these processes can be determined, ensuring that potential memory limit problems can be discovered and resolved in a timely manner, ensuring stable operation and performance optimization of the job.
[0081] In a possible implementation, for each Flink job obtained, on an operating system, which may be Linux, for example, each cluster node may be traversed according to the job identifiers of all Flink jobs that need to detect whether physical memory usage exceeds the limit, that is, applicationId, so as to obtain the process number, process type name and process command details of all nodes of each applicationId according to the command.
[0082] According to the process number of each process, the system file on the operating system is read. For example, the Linux system file can be / proc / ${pid} / status, where the ${pid} variable is the process number of the process. According to the process number in the system file, the parent process number of the process, the anonymous physical memory RSSAnon, the file physical memory RssFile and the shared memory RssShmem values are further collected to determine the used physical memory of all processes of the Flink job.
[0083] S103: Determine the total physical memory used by the Flink job according to the physical memory used by the multiple processes.
[0084] The total used physical memory may be obtained by adding up the used physical memory of all processes in each Flink job to be detected, for example.
[0085] This step determines the total physical memory used by the Flink job based on the physical memory used by multiple processes on the Flink job. The purpose is to fully understand and monitor the resource consumption of the entire job, ensure the reasonable allocation and efficient use of resources, and timely detect whether the physical memory used by the Flink job exceeds the limit.
[0086] S104: Determine the allocated physical memory of the Flink job, and determine whether the total physical memory of the Flink job exceeds the limit according to the allocated physical memory and the total used physical memory.
[0087] The physical memory allocated to the Flink job refers to the total amount of physical memory allocated to the Flink job in the Flink cluster, including the sum of the physical memory allocated to the job by all processes.
[0088] Determine the allocated physical memory of the Flink job and judge whether the total physical memory of the Flink job exceeds the limit based on the allocated physical memory and the total used physical memory. This is to ensure that the job does not suffer performance degradation or job failure due to insufficient memory during operation.
[0089] In a possible implementation, the physical memory used by all related processes can be collected and summarized to obtain the total physical memory used. Then the allocated physical memory of the Flink job is obtained, that is, the total physical memory allocated for the job. Finally, the total used physical memory is compared with the allocated physical memory. If the total used physical memory exceeds the allocated physical memory, it is determined that the physical memory of the job exceeds the limit. At this time, corresponding measures can be taken, such as increasing memory allocation, optimizing code, or reducing parallelism, to solve the problem of memory limit.
[0090] The embodiment of the present application provides a method for determining the physical memory overlimit of a Flink job, which determines the job identifier corresponding to the Flink job to be detected, obtains multiple processes contained in each Flink job to be detected, thereby obtaining the used physical memory of each process, and then determines the total used physical memory of each Flink job to be detected by calculation, and finally compares the total used physical memory of each Flink job to be detected with its corresponding allocated physical memory to determine whether the used physical memory of each Flink job to be detected exceeds the limit. The method realizes the determination of whether the physical memory of the Flink job has an overlimit problem, thereby avoiding waste of resources and improving the reliability and processing efficiency of the job.
[0091] Figure 2 A method for determining physical memory limit excess of a Flink job provided in an embodiment of the present application Figure 2 This embodiment is in Figure 1Based on the embodiment, the method for determining the physical memory limit of a Flink job is described in detail. Figure 1 As shown, the method for determining physical memory limit excess of a Flink job provided in this embodiment includes:
[0092] S201: Determine a job identifier corresponding to a Flink job.
[0093] Among them, step S201 is similar to the above-mentioned step S101, and will not be repeated here in this application.
[0094] S202: Determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine physical memory used by the multiple processes.
[0095] Among them, step S202 is similar to the above-mentioned step S102, and this application will not repeat them here.
[0096] S203: For any one process among the multiple processes, determine the process information of the process, where the process information includes: a process number.
[0097] For any process of the Flink job to be detected, you can search for process information related to the applicationId according to the applicationId corresponding to each Flink job. The specific search command can be as follows:
[0098] ps -aux | grep ${applicationId} | grep -v grep | awk '{print $2 ":" $11 ":" $12}
[0099] Among them, $2 is the process ID of the corresponding process, $11 is the corresponding process type name, $12 is the corresponding process command details, ps -aux is to list the detailed information of all processes in the system, grep ${applicationId} is to filter out the lines containing the specified applicationId from the process list, grep -v grep is to exclude the lines containing grep itself to avoid mismatching. awk '{print $2 ":" $11 ":" $12} is to extract and format the specific fields from the filtered lines. Through this command, the final output format is the process ID of the process.
[0100] Optionally, from the obtained process command details field, the string can also be split according to " / " to obtain multiple arrays and take the data with the -2 subscript, that is, the second to last data in the array. If the data contains the container string, the data is used as the process name of the current process corresponding to the Flink job, that is, containerid. Based on the process number, physical memory usage, parent process number, anonymous physical memory usage, file physical memory usage, and shared memory corresponding to each process, the collected information about the use of physical memory for each process of each Flink job can be recorded in the database Mysql table 1 for subsequent query and processing.
[0101] S204: Determine the system file corresponding to the process number, and determine the anonymous physical memory, file physical memory and shared memory corresponding to the system file.
[0102] Among them, according to the process number of each process, by reading the system file corresponding to the process number on the operating system, the system file is usually located in the / proc directory, thereby collecting the parent process number of the process, anonymous physical memory RSSAnon, file physical memory RssFile and shared memory RssShmem corresponding values.
[0103] In one possible way, the parent process number can be obtained by reading the / proc / ${pid} / status file, and the detailed information of anonymous physical memory RSSAno, file physical memory RssFile, and shared memory RssShmem can be obtained by reading the / proc / ${pid} / smaps or / proc / ${pid} / stat file.
[0104] S205: The sum of the anonymous physical memory, the file physical memory and the shared memory is used as the physical memory used by the process.
[0105] Among them, based on anonymous physical memory, file physical memory and shared memory, the physical memory VmRSS used by each process can be calculated using the following formula:
[0106] VmRSS=RSSAnon+RssFile+RssShmem
[0107] The physical memory VmRSS used here refers to the size of the physical memory actually occupied by the application.
[0108] S206: Taking the sum of the physical memory used by the multiple processes as the total physical memory used by the Flink job.
[0109] The multiple processes of a Flink job include: multiple task processes and one job process. This solution does not distinguish between task processes and job processes. It only needs to add up the physical memory of all processes under the same applicationId to get the total physical memory used by the current Flink job.
[0110] S207: Determine the maximum parallelism of the Flink job according to the job identifier, and determine the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process in the Flink job, and the number of the task processes.
[0111] Among them, the maximum degree of parallelism is used to define the maximum number of parallel tasks that a job can achieve during runtime. It determines how many parallel task units the job can be divided into during runtime, which in turn affects the execution efficiency and resource utilization of the job.
[0112] According to the job identifier of the Flink job, that is, applicationId, you can use the Flink REST API to obtain the job ID of the current Flink job, that is, jobId, from http: / / *** / proxy / applicationId / v1 / jobs. Then, based on http: / / *** / proxy / applicationId / v1 / jobs / jobId / , you can obtain detailed information about the job, including the maximum parallelism of the current job. At the same time, you can also use the Flink REST API to obtain the size of the allocated physical memory of each task process in the Flink job and the number of the task processes, as well as the size of the allocated physical memory of a job process included in the job, from http: / / *** / proxy / applicationId / v1 / jobmanager / config.
[0113] Optionally, based on the job name, maximum parallelism, size of physical memory allocated to each task process, number of task processes, and size of physical memory allocated to the job process of the Flink job, the collected physical memory allocation information of each Flink job can be recorded in the database Mysql table 2 for subsequent query and processing.
[0114] S208: Calculate and process based on the maximum parallelism, the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process, and the number of task processes to obtain the allocated physical memory of the Flink job.
[0115] Among them, based on the maximum degree of parallelism, the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process, and the number of task processes, the allocated physical memory of each process can be calculated using the following formula:
[0116] size_max=taskmanagerSizes*(parallelism / taskslots)+jobmanagerSizes
[0117] Among them, size_max is the size of physical memory allocated to each process, taskmanagerSizes is the size of physical memory allocated to each task process, parallelism is the maximum degree of parallelism, parallelism / taskslots is the number of task processes, taskslots refers to the task slots in the task manager, and jobmanagerSizes is the size of physical memory allocated to the job process.
[0118] S209: Determine whether the total used physical memory is greater than the allocated physical memory; if so, execute step S210, if not, execute step S211.
[0119] This step determines whether the total physical memory is greater than the allocated physical memory in order to determine whether the total physical memory of the Flink job exceeds the limit.
[0120] Optionally, you can also use the following method to determine whether the total physical memory of the Flink job exceeds the limit. For example, subtract the total physical memory used by the Flink job from the allocated physical memory of the Flink job to obtain the remaining physical memory of the Flink job, and then determine the remaining physical memory of the Flink job. If the remaining physical memory is not less than zero, the total physical memory usage of the Flink job does not exceed the limit; if the remaining physical memory is less than zero, the total physical memory usage of the Flink job exceeds the limit.
[0121] Optionally, based on the job name, allocated physical memory, total used physical memory, and remaining physical memory of the Flink job, the calculated physical memory limit information of each Flink job can be recorded in the database MySQL table 3 as the final conclusion record table for subsequent query.
[0122] S210: The total physical memory of the Flink job exceeds the limit.
[0123] Among them, when the total physical memory used by the Flink job is greater than the allocated physical memory of the Flink job, it indicates that the total physical memory of the Flink job has exceeded the limit. At this time, users can take measures such as increasing memory allocation, optimizing code and data processing logic, reducing parallelism, setting monitoring and alarm mechanisms, and optimizing checkpoint and savepoint configurations to avoid potential performance problems and job failures and ensure stable operation and efficient execution of jobs.
[0124] S211: The total physical memory of the Flink job does not exceed the limit.
[0125] Among them, if the total physical memory used by the Flink job is not greater than the allocated physical memory of the Flink job, it indicates that the total physical memory of the Flink job is within the limit. At this time, it can be considered that the memory usage of the job is within the controllable range and the system is running stably. However, it is still recommended to continuously monitor the memory usage to ensure the rational use of resources and optimize according to actual needs to further improve the performance and stability of the job.
[0126] The embodiment of the present application provides a method for determining the physical memory overlimit of a Flink job, by determining the job identifier corresponding to the Flink job, further obtaining multiple processes corresponding to the Flink job, and then obtaining corresponding information and calculating the physical memory used by each process according to the process number of each process, and adding them up to obtain the total physical memory used by the Flink job, and then determining the allocated physical memory of the Flink job according to the job identifier, comparing the total physical memory used with the allocated physical memory, and judging and determining whether the total physical memory usage of the Flink job exceeds the limit. This method realizes the determination of the physical memory overlimit of the Flink job, thereby avoiding potential performance problems and job failures, and ensuring the stable operation and efficient execution of the job.
[0127] Figure 3 This is a schematic diagram of the structure of the device for determining the physical memory limit of a Flink job provided by this application. Figure 3 As shown, the physical memory limit determination device 300 of a Flink job provided in this embodiment includes:
[0128] A determination module 301 is used to determine a job identifier corresponding to a Flink job;
[0129] The determination module 301 is further used to determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine physical memory used by the multiple processes, wherein the multiple processes include: multiple task processes and one job process;
[0130] The determination module 301 is further configured to determine the total physical memory used by the Flink job according to the physical memory used by the multiple processes;
[0131] The determination module 301 is further used to determine the allocated physical memory of the Flink job;
[0132] The judgment module 302 is used to judge whether the total physical memory of the Flink job exceeds the limit according to the allocated physical memory and the total used physical memory.
[0133] Optionally, the device further includes: a processing module 303;
[0134] The determination module 301 is further used to determine, for any one of the multiple processes, process information of the process, the process information including: a process number; determining a system file corresponding to the process number, and determining anonymous physical memory usage, file physical memory usage, and shared memory corresponding to the system file;
[0135] The processing module 303 is used to use the sum of the anonymous physical memory, the file physical memory and the shared memory as the physical memory used by the process.
[0136] Optionally, the processing module 303 is further configured to use the sum of the used physical memories of the multiple processes as the total used physical memory of the Flink job.
[0137] Optionally, the determination module 301 is further configured to determine a character string corresponding to the process according to the process command information;
[0138] The processing module 303 is further used to segment the string to obtain multiple arrays; use the array at a preset position in the multiple arrays as the process name of the process; and generate a first physical memory table corresponding to the Flink job according to the process names corresponding to the multiple processes and the used physical memory.
[0139] Optionally, the processing module 303 is further used to perform data analysis and processing according to the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process to obtain the allocated physical memory of the Flink job;
[0140] The determination module 302 is further configured to compare the allocated physical memory with the total physical memory to determine whether the total physical memory of the Flink job exceeds a limit.
[0141] Optionally, the determination module 301 is further used to determine the maximum parallelism of the Flink job according to the job identifier, and determine the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process in the Flink job and the number of the task processes;
[0142] The processing module 303 is further used to perform calculations based on the maximum degree of parallelism, the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process, and the number of task processes to obtain the allocated physical memory of the Flink job.
[0143] Optionally, the determination module 301 is further used to determine that the total physical memory of the Flink job exceeds the limit when the total physical memory is greater than the allocated physical memory; and to determine that the total physical memory of the Flink job does not exceed the limit when the total physical memory is less than or equal to the allocated physical memory.
[0144] This embodiment provides a device for determining physical memory limit excess of a Flink job, which can execute the method for determining physical memory limit excess of a Flink job provided by the above method embodiment. Its implementation principle and technical effect are similar, and are not described in detail in this embodiment.
[0145] Figure 4 A schematic diagram of the structure of a device for determining physical memory limit overrun for a Flink job provided in this application. Figure 4 As shown, the present application provides a device for determining a physical memory limit excess of a Flink job. The device 400 for determining a physical memory limit excess of a Flink job includes: a receiver 401 , a transmitter 402 , a processor 403 , and a memory 404 .
[0146] Receiver 401, used for receiving instructions and data;
[0147] A transmitter 402, used for sending instructions and data;
[0148] Memory 404, for storing computer-executable instructions;
[0149] The processor 403 is configured to execute the computer-executable instructions stored in the memory 404 to implement the steps performed by the method for determining the physical memory limit of a Flink job in the above embodiment. For details, please refer to the relevant description in the embodiment of the method for determining the physical memory limit of a Flink job.
[0150] Optionally, the memory 404 may be independent or integrated with the processor 403 .
[0151] When the memory 404 is independently provided, the electronic device further includes a bus for connecting the memory 404 and the processor 403 .
[0152] The present application also provides a computer storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, a method for determining a physical memory limit of a Flink job as executed by the physical memory limit determination device for a Flink job as described above is implemented.
[0153] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical nodes; for example, a physical node may have multiple functions, or a function or step may be performed by several physical nodes in cooperation. Some physical nodes or all physical nodes may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0154] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0155] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for determining physical memory limit excess of a Flink job, characterized in that: include: Determine the job identifier corresponding to the Flink job; According to the job identifier, determine multiple processes corresponding to the Flink job, and determine physical memory used by the multiple processes, wherein the multiple processes include: multiple task processes and one job process; Determine the total physical memory used by the Flink job according to the physical memory used by the multiple processes; Determine the allocated physical memory of the Flink job, and determine whether the total physical memory of the Flink job exceeds a limit according to the allocated physical memory and the total used physical memory.
2. The method according to claim 1, characterized in that: The determining of the physical memory usage of the multiple processes includes: For any one of the multiple processes, determine process information of the process, the process information including: a process number; Determine the system file corresponding to the process number, and determine the anonymous physical memory, file physical memory and shared memory corresponding to the system file; The sum of the anonymous physical memory, the file physical memory and the shared memory is taken as the physical memory used by the process.
3. The method according to claim 2, characterized in that Determining the total physical memory used by the Flink job according to the physical memory used by the multiple processes includes: The sum of the physical memories used by the multiple processes is taken as the total physical memory used by the Flink job.
4. The method according to claim 2, characterized in that: The process information further includes: process command information, and the method further includes: Determine, according to the process command information, a character string corresponding to the process; Segment the character string to obtain multiple arrays; Using an array at a preset position among the multiple arrays as the process name of the process; A first physical memory table corresponding to the Flink job is generated according to the process names and used physical memory corresponding to the multiple processes.
5. The method according to claim 1, characterized in that The allocated physical memory includes: the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process; the determined allocated physical memory of the Flink job, and judging whether the total physical memory of the Flink job exceeds the limit according to the allocated physical memory and the total used physical memory, include: Perform data analysis and processing according to the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process to obtain the allocated physical memory of the Flink job; The allocated physical memory is compared with the total physical memory to determine whether the total physical memory of the Flink job exceeds the limit.
6. The method according to claim 5, characterized in that The performing data analysis and processing according to the allocated physical memory of the multiple task processes and the allocated physical memory of the one job process to obtain the allocated physical memory of the Flink job includes: According to the job identifier, determine the maximum parallelism of the Flink job, and determine the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process in the Flink job, and the number of the task processes; The allocated physical memory of the Flink job is obtained by performing calculation based on the maximum parallelism, the size of the allocated physical memory of the job process, the size of the allocated physical memory of each task process, and the number of the task processes.
7. The method according to claim 6, characterized in that The comparing the allocated physical memory with the total physical memory to determine whether the total physical memory of the Flink job exceeds the limit includes: If the total used physical memory is greater than the allocated physical memory, the total physical memory of the Flink job exceeds the limit; If the total used physical memory is less than or equal to the allocated physical memory, the total physical memory of the Flink job does not exceed the limit.
8. A device for determining physical memory limit excess of a Flink job, characterized in that: The device comprises: The determination module is used to determine the job identifier corresponding to the Flink job; The determination module is further used to determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine physical memory used by the multiple processes, wherein the multiple processes include: multiple task processes and one job process; The determination module is further configured to determine the total physical memory used by the Flink job according to the physical memory used by the multiple processes; The determination module is further used to determine the allocated physical memory of the Flink job; The judgment module is used to judge whether the total physical memory of the Flink job exceeds the limit according to the allocated physical memory and the total used physical memory.
9. A device for determining physical memory limit excess of a Flink job, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for determining physical memory excess of a Flink job according to any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which, when executed by a processor, are used to implement the method for determining physical memory excess of a Flink job according to any one of claims 1 to 7.