Flink job virtual memory utilization rate determination method and device, equipment and storage medium

Abstract

The invention provides an Flink operation virtual memory utilization rate determination method and device, equipment and a storage medium, and can be applied to the technical field of Flink operation virtual memory utilization. According to the method, a job identifier of an Flink job is obtained, a virtual memory used by each process in the Flink job is determined, so that the total used virtual memory of the Flink job is determined, according to the job identifier of the Flink job, a distributed virtual memory of the Flink job can also be determined, the total used virtual memory and the distributed virtual memory are calculated, and the total used virtual memory and the distributed virtual memory are calculated. And finally, determining the virtual memory utilization rate of the Flink operation. According to the method, the Flink job virtual memory utilization rate is determined, resource waste is avoided, reasonable allocation and efficient utilization of resources are guaranteed, and job stability is improved.

Description

Technical Field

[0001] The present application relates to the technical field of Flink job virtual memory usage, and in particular to a method, apparatus, device and storage medium for determining a Flink job virtual memory usage rate. Background Art

[0002] In recent years, with the increasing number of Flink jobs, its stable operation has become particularly important. The factors that affect the stable operation of Flink jobs mainly include parallelism settings and memory management. As the number of Flink jobs continues to increase, the resources occupied by the jobs will also increase, and the problem of insufficient memory will gradually become prominent.

[0003] The usage of virtual memory directly affects the normal operation of Flink jobs. If the usage of virtual memory exceeds 100%, the Flink job will crash directly. However, the existing Flink job performance collection only involves simple analysis of heap memory to optimize job performance, and does not involve methods for collecting virtual memory usage of Flink jobs running on Yarn clusters.

[0004] In summary, the current work on collecting and recording the virtual memory usage of Flink jobs running on a Yarn cluster is still imperfect. Therefore, the present invention provides a method for determining the virtual memory usage of a Flink job. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for determining the virtual memory usage of a Flink job, so as to solve the problem that the collection and recording of the virtual memory usage of a Flink ON YARN job in the existing monitoring system is not perfect.

[0006] In a first aspect, the present application provides a method for determining a virtual memory usage rate of a Flink job, the method comprising:

[0007] Obtain a job identifier corresponding to a Flink job, and determine multiple processes corresponding to the Flink job according to the job identifier, and determine virtual memory used by the multiple processes, wherein the multiple processes include: a job process and multiple task processes;

[0008] Determine the total virtual memory used by the Flink job according to the virtual memory used by the multiple processes;

[0009] Determine, according to the job identifier, a first allocated physical memory corresponding to a plurality of task processes in the Flink job and a second allocated physical memory corresponding to the task process;

[0010] Based on the multiple first allocated physical memories and the second allocated physical memories, the allocated virtual memory of the Flink job is determined, and according to the allocated virtual memory and the total used virtual memory, the virtual memory usage rate of the Flink job is determined.

[0011] Optionally, determining the virtual memory usage of the multiple processes includes:

[0012] For any one of the multiple processes, determine process information of the process, the process information including: a process number;

[0013] Determine the system file corresponding to the process number, and determine the used virtual memory corresponding to the system file;

[0014] The used virtual memory is used as the used virtual memory corresponding to the process.

[0015] Optionally, determining the total used virtual memory of the Flink job according to the used virtual memories of the multiple processes includes:

[0016] The sum of the virtual memories used by the multiple processes is taken as the total virtual memory used by the Flink job.

[0017] Optionally, determining, according to the job identifier, a first allocated physical memory corresponding to a plurality of task processes in the Flink job and a second allocated physical memory corresponding to the task process includes:

[0018] According to the job identifier, the maximum parallelism of the Flink job is determined through the API interface, and the first allocated physical memory corresponding to each task process in the Flink job, the second allocated physical memory corresponding to the job process in the Flink job, and the number of task slots are determined.

[0019] Optionally, determining the allocated virtual memory of the Flink job based on the multiple first allocated physical memories and the second allocated physical memories, and determining the virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory includes:

[0020] Performing data analysis based on the plurality of first allocated physical memories and the second allocated physical memories to determine a first allocated virtual memory corresponding to each task process and a second allocated virtual memory corresponding to the job process;

[0021] Perform calculation processing according to the maximum degree of parallelism, the number of task slots, the first allocated virtual memory, and the second allocated virtual memory to determine the allocated virtual memory of the Flink job;

[0022] A virtual memory usage rate of the Flink job is determined according to the allocated virtual memory and the total used virtual memory.

[0023] Optionally, determining the virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory includes:

[0024] The quotient of the total used virtual memory and the allocated virtual memory is used as the virtual memory usage of the Flink job.

[0025] Optionally, after taking the used virtual memory as the used virtual memory corresponding to the process, the method further includes:

[0026] Generate a first used virtual memory table corresponding to the Flink job according to the job identifier, the process number, and the used virtual memory of the multiple processes;

[0027] Based on a preset time period, the data in the virtual memory table is written into the Doris database according to a preset storage strategy to obtain a second virtual memory table, wherein the preset storage strategy is tiered processing of hot and cold data, the cold data is used to represent data with low access frequency, and the hot data is used to represent data with high access frequency.

[0028] In a second aspect, the present application provides a device for determining a virtual memory usage rate of a Flink job, the device comprising:

[0029] The acquisition module is used to obtain the job identifier corresponding to the Flink job;

[0030] A determination module, configured to determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine virtual memory usage of the multiple processes, wherein the multiple processes include: a job process and multiple task processes;

[0031] The determination module is further used to determine the total used virtual memory of the Flink job according to the used virtual memory of the multiple processes; determine, according to the job identifier, a first allocated physical memory corresponding to a plurality of task processes in the Flink job and a second allocated physical memory corresponding to the task process; determine the allocated virtual memory of the Flink job based on the plurality of first allocated physical memories and the second allocated physical memory, and determine the virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory.

[0032] Optionally, the device further comprises: a processing module;

[0033] The determination module is further used to determine, for any one of the multiple processes, process information of the process, the process information including: a process number; determining a system file corresponding to the process number, and determining a used virtual memory corresponding to the system file;

[0034] The processing module is further configured to use the used virtual memory as the used virtual memory corresponding to the process.

[0035] Optionally, the processing module is further configured to use the sum of the used virtual memories of the multiple processes as the total used virtual memory of the Flink job.

[0036] Optionally, the determination module is also used to determine the maximum parallelism of the Flink job through the API interface according to the job identifier, and determine the first allocated physical memory corresponding to each task process in the Flink job, the second allocated physical memory corresponding to the job process in the Flink job, and the number of task slots.

[0037] Optionally, the determination module is further used to perform data analysis and processing based on the multiple first allocated physical memories and the second allocated physical memories to determine the first allocated virtual memory corresponding to each task process and the second allocated virtual memory corresponding to the job process; perform calculation processing according to the maximum degree of parallelism, the number of task slots, the first allocated virtual memory, and the second allocated virtual memory to determine the allocated virtual memory of the Flink job; determine the virtual memory usage rate of the Flink job based on the allocated virtual memory and the total used virtual memory.

[0038] Optionally, the processing module is further configured to use a quotient of the total used virtual memory and the allocated virtual memory as the virtual memory usage rate of the Flink job.

[0039] Optionally, the processing module is further used to generate a first used virtual memory table corresponding to the Flink job according to the job identifier, the process number, and the used virtual memory of the multiple processes; based on a preset time period, write the data in the virtual memory table into a Doris database according to a preset storage strategy to obtain a second used virtual memory table, wherein the preset storage strategy is tiered processing of hot and cold data, the cold data is used to represent data with low access frequency, and the hot data is used to represent data with high access frequency.

[0040] In a third aspect, the present application provides a device for determining virtual memory usage of a Flink job, including:

[0041] Memory;

[0042] processor;

[0043] Wherein, the memory stores computer-executable instructions;

[0044] The processor executes the computer-executable instructions stored in the memory to implement the method for determining the virtual memory usage rate of a Flink job as described in the first aspect and various possible implementations of the first aspect.

[0045] In a fourth aspect, the present application provides a computer storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method for determining the virtual memory usage rate of a Flink job as described in the first aspect and various possible implementations of the first aspect.

[0046] The present application provides a method for determining the virtual memory usage rate of a Flink job. By obtaining the job identifier of the Flink job, the virtual memory used by each process in the Flink job is determined, thereby determining the total virtual memory used by the Flink job. According to the job identifier of the Flink job, the allocated virtual memory of the Flink job can also be determined. The total virtual memory used and the allocated virtual memory are calculated to finally determine the virtual memory usage rate of the Flink job. The method realizes the determination of the virtual memory usage rate of the Flink job, avoids the waste of resources, ensures the reasonable allocation and efficient use of resources, and improves the stability of the job. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A method for determining virtual memory usage of a Flink job provided in this application Figure 1 ;

[0049] Figure 2 A method for determining virtual memory usage of a Flink job provided in this application Figure 2 ;

[0050] Figure 3 A schematic diagram of the structure of a device for determining virtual memory usage of a Flink job provided in this application;

[0051] Figure 4 A schematic diagram of the structure of a Flink job virtual memory usage determination device provided in this application.

[0052] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0055] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] First, the terms involved in this application are explained:

[0057] Apache Flink: It is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data flow engine used for stateful computing of unbounded and bounded data streams. Flink can run in all common cluster environments and can perform calculations at memory speed and any scale; Flink runtime consists of two types of processes: a job manager (JobManager) and one or more task managers (TaskManager).

[0058] Job Manager: Mainly responsible for scheduling tasks, coordinating checkpoint error recovery, etc. When the client submits the packaged job to the JobManager, the JobManager will assign the job to a TaskManager with resources based on the registered TaskManager resource information, and then start running the job. TaskManager obtains task information from the JobManager, and then uses slot resources to run the task.

[0059] Task Manager: It is a task that executes data flow. A task may have multiple subtasks by setting the degree of parallelism. Each TaskManager runs as an independent JVM process. It is mainly responsible for executing operators in independent threads. The number of operators that can be executed depends on the number of slots specified by each taskManager. Task slot is the smallest resource unit in Flink. If a taskManager has 2 slots, it will allocate 1 / 2 of the memory resources to each slot. Slots in the same taskManager will share network resources and heartbeat information.

[0060] Doris: It is a high-performance, real-time analytical database based on the Massively Parallel Processing (MPP) architecture. It is well-known for its extremely fast and easy-to-use features. It can return query results for massive data in sub-second response time. It can not only support high-concurrency point query scenarios, but also support high-throughput complex analysis scenarios. Based on this, Apache Doris can better meet the use scenarios such as report analysis, ad hoc query, unified data warehouse construction, data lake federated query acceleration, etc. Users can build applications such as user behavior analysis, log retrieval analysis, user portrait analysis, and order analysis on it.

[0061] In recent years, with the increasing number of Flink jobs, its stable operation has become particularly important. The factors that affect the stable operation of Flink jobs mainly include parallelism settings and memory management. As the number of Flink jobs continues to increase, the resources occupied by the jobs will also increase, and the problem of insufficient memory will gradually become prominent.

[0062] The usage of virtual memory directly affects the normal operation of Flink jobs. If the usage of virtual memory exceeds 100%, the Flink job will crash directly. However, the existing Flink job performance collection only involves simple analysis of heap memory to optimize job performance, and does not involve methods for collecting virtual memory usage of Flink jobs running on Yarn clusters.

[0063] In response to the above problems, this application proposes a method for determining the virtual memory usage rate of a Flink job.

[0064] This method obtains the job ID of the Flink job and determines the used virtual memory of each process in the Flink job, thereby determining the total used virtual memory of the Flink job. According to the job ID of the Flink job, the allocated virtual memory of the Flink job can also be determined. The total used virtual memory and the allocated virtual memory are calculated to finally determine the virtual memory usage rate of the Flink job. This method realizes the determination of the virtual memory usage rate of the Flink job, avoids the waste of resources, ensures the reasonable allocation and efficient use of resources, and improves the stability of the job.

[0065] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] Figure 1 This is a process of determining the virtual memory usage rate of a Flink job provided by an embodiment of the present application. Figure 1 .like Figure 1 As shown, the method for determining the virtual memory usage rate of a Flink job provided in this embodiment includes:

[0067] S101: Obtain a job identifier corresponding to a Flink job, and determine multiple processes corresponding to the Flink job according to the job identifier, and determine virtual memory usage of the multiple processes, wherein the multiple processes include: a job process and multiple task processes.

[0068] The job ID is an applicationId for each Flink job, which is the unique ID of each Flink job and can help users and systems manage in the Flink cluster. The job process refers to the Jobmanager process, and the task process refers to the Container process in the Taskmanager node. A Flink job has one Jobmanager process and N container processes of N Taskmanager nodes.

[0069] The purpose of obtaining the job identifier corresponding to the Flink job and determining the multiple processes corresponding to the Flink job based on the job identifier is to accurately monitor and manage the resource usage of the Flink job. Through the job identifier, the specific Flink job can be accurately located, and then the information of all its related processes can be obtained, so as to fully understand the resource usage of each process, especially the usage of virtual memory, which helps to timely discover and solve potential resource bottlenecks and performance problems, and ensure the stable operation and efficient execution of the job. And this step determines the virtual memory used by each process, which can enable the subsequent steps to continue to determine the total virtual memory used by the Flink job.

[0070] In a possible implementation, by connecting to the Stream Park database, all Flink jobs for which virtual memory usage needs to be determined are read, and the applicationId value and job name corresponding to each Flink job are retrieved.

[0071] For each Flink job to be determined, on an operating system, which may be Linux, for example, each cluster node may be traversed according to the applicationId of the Flink job, thereby obtaining the process number, process type name, and process command details of all nodes of each applicationId according to the command.

[0072] Then, the system file on the operating system may be read, for example, the Linux system file, the file path is / proc / ${pid} / status, where the ${pid} variable is the process ID of the process. According to the process ID in the system file, the parent process ID of the process and the used virtual memory are further collected.

[0073] S102: Determine the total virtual memory used by the Flink job according to the virtual memory used by the multiple processes.

[0074] According to the operating principle of Flink jobs, the used virtual memory includes TaskManager and JobManager virtual memory. For the same job, there are N container processes of N Taskmanager nodes, that is, task processes, and one JobManager process, that is, job process.

[0075] This step determines the total virtual memory used by the Flink job through the virtual memory used by multiple processes. Its purpose is to subsequently calculate the virtual memory usage of the Flink job, so as to comprehensively monitor and manage the resource usage of the job, timely discover and solve potential resource bottlenecks and performance problems, and ensure stable operation and efficient execution of the job.

[0076] In a possible implementation, the total virtual memory used by each Flink job can be obtained by adding the virtual memory used by all task processes to the virtual memory used by the job process, that is, the virtual memory used by all processes under the same job identifier is added, which is the total virtual memory used by the current Flink job.

[0077] S103: Determine, according to the job identifier, a first allocated physical memory corresponding to a plurality of task processes in the Flink job and a second allocated physical memory corresponding to the task process.

[0078] The first allocated physical memory may be the size of the allocated physical memory of each task process in the multiple task processes in the Flink job, and the second allocated physical memory may be the size of the allocated physical memory of the job process in the Flink job.

[0079] It is understandable that most of the existing technologies for determining the virtual memory usage of Flink jobs only focus on the virtual memory of TaskManager, i.e., the task process, and ignore the virtual memory of JobManager, i.e., the job process. Therefore, this solution separates the virtual memory of TaskManager and JobManager for calculation, and uses their sum as the total virtual memory used by the Flink job.

[0080] In a possible implementation, for each acquired Flink job, the allocated physical memory of each process may be collected through the REST API interface provided by Flink, and then further configured through Yarn to calculate the allocated virtual memory allocated to the Flink job.

[0081] As you can understand, Apache Flink is an open source stream processing framework that includes cluster deployment modes such as standalone cluster mode, Yarn mode, and Kubernetes. The Flink on Yarn cluster deployment mode involves two open source frameworks, Yarn and Flink. The principle is to rely on Yarn to schedule Flink jobs, and Yarn is responsible for coordinating the allocation of various resources in the cluster to various applications. Flink on Yarn takes advantage of Yarn's resource management and scheduling capabilities and can work well with other components in the Hadoop cluster. When a Flink job runs on Yarn, even if the Flink Application Master fails, the job will continue to execute and be restarted by the Yarn ResourceManager. In addition, Flink can run directly on Yarn without the need for additional adapters or special configuration. Flink on Yarn can also take advantage of Yarn's scalability to process larger data sets by adding more nodes, and can work well with existing Hadoop clusters without the need for additional settings.

[0082] S104: Determine an allocated virtual memory of the Flink job based on the multiple first allocated physical memories and the second allocated physical memories, and determine a virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory.

[0083] The configuration process may be performed through Yarn based on the first allocated physical memory of the multiple task processes of the Flink job and the second allocated physical memory of the job process.

[0084] In one possible implementation, the yarn cluster's virtual memory configuration information enables virtual memory check yarn.nodemanager.vmem-check-enabled by default, and the default virtual memory ratio yarn.nodemanager.vmem-pmem-ratio is 2.1 without any additional modification. That is, the allocated virtual memory size of the job process is 2.1 times the allocated physical memory size of the job process. Similarly, the allocated virtual memory size of the task process is also 2.1 times the allocated physical memory size of the task process. Based on the allocated virtual memory of multiple processes in a Flink job, the allocated virtual memory of the Flink job can be determined. At the same time, the virtual memory usage rate of the Flink job can also be determined based on the allocated virtual memory of the Flink job and the total virtual memory used by the Flink job.

[0085] The embodiment of the present application provides a method for determining the virtual memory usage rate of a Flink job. The method obtains the job identifier of the Flink job, determines multiple processes corresponding to the Flink job, and determines the virtual memory used by each process. By adding them together, the total virtual memory used by the Flink job can be obtained. According to the job identifier of the Flink job, the allocated physical memory of each process in the Flink job can also be determined, thereby obtaining the allocated virtual memory of each process of the Flink job, thereby obtaining the allocated virtual memory of the Flink job, calculating the total virtual memory used and the allocated virtual memory, and finally determining the virtual memory usage rate of the Flink job. The method realizes the determination of the virtual memory usage rate of the Flink job, thereby avoiding the waste of resources and improving the reliability and processing efficiency of the job.

[0086] Figure 2 A method for determining the virtual memory usage of a Flink job provided in an embodiment of the present application Figure 2 This embodiment is in Figure 1 Based on the embodiment, the method for determining the virtual memory usage rate of a Flink job is described in detail. Figure 1 As shown, the method for determining the virtual memory usage rate of a Flink job provided in this embodiment includes:

[0087] S201: Obtain a job identifier corresponding to a Flink job, and determine multiple processes corresponding to the Flink job according to the job identifier.

[0088] Among them, step S201 is similar to the above-mentioned step S101, and this application will not repeat them here.

[0089] S202: For any one process among the multiple processes, determine the process information of the process, where the process information includes: a process number.

[0090] For any process of the Flink job to be detected, you can search for process information related to the applicationId according to the applicationId corresponding to each Flink job. The specific search command can be as follows:

[0091] ps -aux | grep ${applicationId} | grep -v grep | awk '{print $2 ":" $11 ":" $12}

[0092] Among them, $2 is the process ID of the corresponding process, $11 is the corresponding process type name, $12 is the corresponding process command details, ps -aux is to list the detailed information of all processes in the system, grep ${applicationId} is to filter out the lines containing the specified applicationId from the process list, grep -v grep is to exclude the lines containing grep itself to avoid mismatching. awk '{print $2 ":" $11 ":" $12} is to extract and format the specific fields from the filtered lines. Through this command, the final output format is the process ID of the process.

[0093] Optionally, from the obtained process command details field, the string can be split according to " / " to obtain multiple arrays and the data with the -2 subscript is taken, that is, the second to last data in the array. If the data contains the container string, the data is used as the process name of the current process corresponding to the Flink job, that is, containerid.

[0094] S203: Determine the system file corresponding to the process number, and determine the used virtual memory corresponding to the system file.

[0095] According to the process number of each process, the system file corresponding to the process number on the operating system can be read. The system file is usually located in the / proc directory, so as to collect the used virtual memory corresponding to the process.

[0096] Optionally, you can read the Linux system file / proc / ${pid} / status to collect the virtual memory used by the process corresponding to the process ID and the parent process ID.

[0097] S204: Using the used virtual memory as the used virtual memory corresponding to the process.

[0098] The virtual memory used by the process read from the system file based on the process number is the virtual memory used by the process corresponding to the process number.

[0099] Optionally, based on the process number corresponding to each process, the used virtual memory, the parent process number and the job identifier of the Flink job corresponding to each process, the collected information about the use of virtual memory for each process of each Flink job can be recorded in the database Mysql table 1 to generate a first used virtual memory table for subsequent query and processing.

[0100] S205: Taking the sum of the used virtual memories of the multiple processes as the total used virtual memory of the Flink job, wherein the multiple processes include: a job process and multiple task processes.

[0101] Among them, a Flink job has one job process and N task processes of N Taskmanager nodes. In order to calculate the virtual memory used by the Flink job, the virtual memory used by all task processes and job processes in all Flink jobs can be added together. The final result is the total virtual memory used by the Flink job.

[0102] S206: According to the job identifier, determine the maximum parallelism of the Flink job through the API interface, and determine the first allocated physical memory corresponding to each task process in the Flink job, the second allocated physical memory corresponding to the job process in the Flink job, and the number of task slots.

[0103] Among them, the maximum degree of parallelism is used to define the maximum number of parallel tasks that a job can achieve during runtime. It determines how many parallel task units the job can be divided into during runtime, which in turn affects the execution efficiency and resource utilization of the job.

[0104] In a possible implementation, the job ID of the current Flink job, that is, jobId, can be obtained from http: / / *** / proxy / applicationId / v1 / jobs through an API interface, that is, the Flink REST API, according to the job identifier of the Flink job. Then, based on http: / / *** / proxy / applicationId / v1 / jobs / jobId / , detailed information of the job, including the maximum parallelism of the current job, can be obtained. At the same time, the first allocated physical memory of each task process in multiple task processes in the Flink job, the second allocated physical memory of a job process included, and the number of task slots can be obtained from http: / / *** / proxy / applicationId / v1 / jobmanager / config through the Flink REST API.

[0105] S207: Perform data analysis based on the multiple first allocated physical memories and the second allocated physical memories to determine the first allocated virtual memory corresponding to each task process and the second allocated virtual memory corresponding to the job process.

[0106] Among them, the yarn cluster's virtual memory configuration information enables virtual memory check yarn.nodemanager.vmem-check-enabled by default, and the default virtual memory ratio is 2.1, that is, the size of the virtual memory allocated to the job process is 2.1 times the size of the physical memory allocated to the job process. Similarly, the size of the virtual memory allocated to the task process is also 2.1 times the size of the physical memory allocated to the task process. Based on multiple first allocated physical memories and second allocated physical memories in the Flink job, the first allocated virtual memory corresponding to each task process and the second allocated virtual memory corresponding to the job process of the Flink job can be determined.

[0107] Optionally, based on the job name, maximum parallelism, first allocated virtual memory corresponding to each task process, and second allocated virtual memory corresponding to the job process of the Flink job, each piece of collected information about the Flink job allocated virtual memory can be recorded in the database Mysql table 2 to generate a first allocated virtual memory table for subsequent query and processing.

[0108] S208: Perform calculation processing according to the maximum parallelism, the number of task slots, the first allocated virtual memory, and the second allocated virtual memory to determine the allocated virtual memory of the Flink job.

[0109] Among them, based on the maximum parallelism, the number of task slots, the size of the first allocated virtual memory of each task process, and the second allocated virtual memory of the job process, the allocated virtual memory of the Flink job can be calculated and determined by the following formula:

[0110] virtual_size_max=virtual_taskmanagerSizes*(parallelism / taskslots)+virtual_jobmanagerSizes

[0111] Among them, virtual_size_max is the size of virtual memory allocated for the Flink job, virtual_taskmanagerSizes is the size of the first virtual memory allocated for each task process, parallelism is the maximum degree of parallelism, parallelism / taskslots is the number of task processes, taskslots refers to the number of task slots in the task manager, and virtual_jobmanagerSizes is the size of the second virtual memory allocated for the job process.

[0112] S209: Taking the quotient of the total used virtual memory and the allocated virtual memory as the virtual memory usage rate of the Flink job.

[0113] The virtual memory usage of a Flink job can be determined by dividing the total virtual memory used by the allocated virtual memory. This ratio reflects the proportion of virtual memory actually used by the job to the allocated virtual memory, thereby achieving the purpose of comprehensively monitoring and managing the resource usage of the job.

[0114] Optionally, based on the job name, allocated virtual memory, total used virtual memory, and virtual memory usage of the Flink job, the calculated virtual memory usage information of each Flink job can be recorded in the database MySQL table 3 as the final conclusion record table for subsequent query.

[0115] In a possible implementation, due to the large amount of collected data, in order to facilitate recording and analysis, data recording can be done by combining the RTS database and the Doris database. For example, the data of the first virtual memory table used and the first virtual memory table allocated can be synchronized to the Doris database in a preset period, such as 7 days later, to form report data for subsequent intermediate process tracking and analysis. That is, in the RTS database of the hot data collected in the past 7 days, the data from 7 days ago is stored in the Doris database. In addition, the Doris database sets the detailed storage mode and the hot and cold data stratification, and the data from 30 days ago is moved to the object storage as cold data, thereby improving the subsequent analysis efficiency of the collected virtual memory usage and detailed data. While synchronizing the data to the Doris database, the data in the first virtual memory table used and the first virtual memory table allocated that have been written into Doris can also be deleted.

[0116] Understandably, the Doris database can set up hot and cold data tiering, for example, specifying a hot data retention period of 30 days, after which the data is moved and stored, thereby achieving the purpose of cost savings. Among them, the high frequency of hot data access requires a fast response.

[0117] The embodiment of the present application provides a method for determining the virtual memory usage of a Flink job. By obtaining the job identifier of the Flink job, the virtual memory used by multiple task processes and one job process in the Flink job is determined, thereby determining the total virtual memory used by the Flink job. Then, according to the job identifier of the Flink job, the allocated physical memory of the Flink job can be determined. Since the virtual memory is 2.1 times the physical memory, the allocated virtual memory of the Flink job can be obtained. The total used virtual memory is divided by the allocated virtual memory, and the quotient is the virtual memory usage of the Flink job. The method realizes the determination of the virtual memory usage of the Flink job, thereby avoiding potential performance problems and job failures, and ensuring the stable operation and efficient execution of the job.

[0118] Figure 3 This is a schematic diagram of the structure of a Flink job virtual memory usage determination device provided by this application. Figure 3 As shown, the Flink job virtual memory usage determination device 300 provided in this embodiment includes:

[0119] The acquisition module 301 is used to obtain the job identifier corresponding to the Flink job;

[0120] A determination module 302 is used to determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine virtual memory usage of the multiple processes, wherein the multiple processes include: a job process and multiple task processes;

[0121] The determination module 301 is further used to determine the total used virtual memory of the Flink job according to the used virtual memory of the multiple processes; determine the first allocated physical memory corresponding to the multiple task processes in the Flink job and the second allocated physical memory corresponding to the task process according to the job identifier; determine the allocated virtual memory of the Flink job based on the multiple first allocated physical memories and the second allocated physical memory, and determine the virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory.

[0122] Optionally, the device further includes: a processing module 303;

[0123] The determination module 302 is further configured to determine, for any one of the multiple processes, process information of the process, the process information including: a process number; determining a system file corresponding to the process number, and determining a used virtual memory corresponding to the system file;

[0124] The processing module 303 is further configured to use the used virtual memory as the used virtual memory corresponding to the process.

[0125] Optionally, the processing module 303 is further configured to use the sum of the used virtual memories of the multiple processes as the total used virtual memory of the Flink job.

[0126] Optionally, the determination module 302 is also used to determine the maximum parallelism of the Flink job through the API interface according to the job identifier, and determine the first allocated physical memory corresponding to each task process in the Flink job, the second allocated physical memory corresponding to the job process in the Flink job, and the number of task slots.

[0127] Optionally, the determination module 302 is further used to perform data analysis and processing based on the multiple first allocated physical memories and the second allocated physical memories to determine the first allocated virtual memory corresponding to each task process and the second allocated virtual memory corresponding to the job process; perform calculation processing according to the maximum degree of parallelism, the number of task slots, the first allocated virtual memory, and the second allocated virtual memory to determine the allocated virtual memory of the Flink job; determine the virtual memory usage rate of the Flink job based on the allocated virtual memory and the total used virtual memory.

[0128] Optionally, the processing module 303 is further configured to use a quotient of the total used virtual memory and the allocated virtual memory as the virtual memory usage rate of the Flink job.

[0129] Optionally, the processing module 303 is further used to generate a first used virtual memory table corresponding to the Flink job according to the job identifier, the process number, and the used virtual memory of the multiple processes; based on a preset time period, write the data in the virtual memory table into the Doris database according to a preset storage strategy to obtain a second used virtual memory table, wherein the preset storage strategy is tiered processing of hot and cold data, the cold data is used to represent data with low access frequency, and the hot data is used to represent data with high access frequency.

[0130] This embodiment provides a device for determining physical memory limit excess of a Flink job, which can execute the method for determining physical memory limit excess of a Flink job provided by the above method embodiment. Its implementation principle and technical effect are similar, and are not described in detail in this embodiment.

[0131] Figure 4 A schematic diagram of the structure of a Flink job virtual memory usage determination device provided in this application. Figure 4 As shown, the present application provides a Flink job virtual memory usage determination device, and the Flink job virtual memory usage determination device 400 includes: a receiver 401, a transmitter 402, a processor 403 and a memory 404.

[0132] Receiver 401, used for receiving instructions and data;

[0133] A transmitter 402, used for sending instructions and data;

[0134] Memory 404, for storing computer-executable instructions;

[0135] The processor 403 is used to execute the computer-executable instructions stored in the memory 404 to implement the various steps performed by the method for determining the virtual memory usage rate of a Flink job in the above embodiment. For details, please refer to the relevant description in the above embodiment of the method for determining the virtual memory usage rate of a Flink job.

[0136] Optionally, the memory 404 may be independent or integrated with the processor 403 .

[0137] When the memory 404 is independently provided, the electronic device further includes a bus for connecting the memory 404 and the processor 403 .

[0138] The present application also provides a computer storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, a Flink job virtual memory usage determination method as performed by the Flink job virtual memory usage determination device described above is implemented.

[0139] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of virtual nodes; for example, a virtual node may have multiple functions, or a function or step may be performed by several virtual nodes in cooperation. Some virtual nodes or all virtual nodes may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0140] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0141] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for determining virtual memory usage of a Flink job, characterized in that: include: Obtain a job identifier corresponding to a Flink job, and determine multiple processes corresponding to the Flink job according to the job identifier, and determine virtual memory used by the multiple processes, wherein the multiple processes include: a job process and multiple task processes; Determine the total virtual memory used by the Flink job according to the virtual memory used by the multiple processes; Determine, according to the job identifier, a first allocated physical memory corresponding to a plurality of task processes in the Flink job and a second allocated physical memory corresponding to the task process; Based on the multiple first allocated physical memories and the second allocated physical memories, the allocated virtual memory of the Flink job is determined, and according to the allocated virtual memory and the total used virtual memory, the virtual memory usage rate of the Flink job is determined.

2. The method according to claim 1, characterized in that Determining the virtual memory usage of the multiple processes includes: For any one of the multiple processes, determine process information of the process, the process information including: a process number; Determine the system file corresponding to the process number, and determine the used virtual memory corresponding to the system file; The used virtual memory is used as the used virtual memory corresponding to the process.

3. The method according to claim 2, characterized in that Determining the total used virtual memory of the Flink job according to the used virtual memory of the multiple processes includes: The sum of the virtual memories used by the multiple processes is taken as the total virtual memory used by the Flink job.

4. The method according to claim 1, characterized in that: The determining, according to the job identifier, a first allocated physical memory corresponding to a plurality of task processes in the Flink job and a second allocated physical memory corresponding to the task process includes: According to the job identifier, the maximum parallelism of the Flink job is determined through the API interface, and the first allocated physical memory corresponding to each task process in the Flink job, the second allocated physical memory corresponding to the job process in the Flink job, and the number of task slots are determined.

5. The method according to claim 1, characterized in that The step of determining the allocated virtual memory of the Flink job based on the plurality of first allocated physical memories and the second allocated physical memories, and determining the virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory includes: Performing data analysis based on the plurality of first allocated physical memories and the second allocated physical memories to determine a first allocated virtual memory corresponding to each task process and a second allocated virtual memory corresponding to the job process; Perform calculation processing according to the maximum degree of parallelism, the number of task slots, the first allocated virtual memory, and the second allocated virtual memory to determine the allocated virtual memory of the Flink job; A virtual memory usage rate of the Flink job is determined according to the allocated virtual memory and the total used virtual memory.

6. The method according to claim 1, characterized in that The determining the virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory includes: The quotient of the total used virtual memory and the allocated virtual memory is used as the virtual memory usage of the Flink job.

7. The method according to claim 2, characterized in that After using the used virtual memory as the used virtual memory corresponding to the process, the method further includes: Generate a first used virtual memory table corresponding to the Flink job according to the job identifier, the process number, and the used virtual memory of the multiple processes; Based on a preset time period, the data in the virtual memory table is written into the Doris database according to a preset storage strategy to obtain a second virtual memory table, wherein the preset storage strategy is tiered processing of hot and cold data, the cold data is used to represent data with low access frequency, and the hot data is used to represent data with high access frequency.

8. A device for determining virtual memory usage of a Flink job, characterized in that: The device comprises: The acquisition module is used to obtain the job identifier corresponding to the Flink job; A determination module, configured to determine, according to the job identifier, multiple processes corresponding to the Flink job, and determine virtual memory usage of the multiple processes, wherein the multiple processes include: a job process and multiple task processes; The determination module is further used to determine the total used virtual memory of the Flink job according to the used virtual memory of the multiple processes; determine, according to the job identifier, a first allocated physical memory corresponding to a plurality of task processes in the Flink job and a second allocated physical memory corresponding to the task process; determine the allocated virtual memory of the Flink job based on the plurality of first allocated physical memories and the second allocated physical memory, and determine the virtual memory usage rate of the Flink job according to the allocated virtual memory and the total used virtual memory.

9. A device for determining virtual memory usage of a Flink job, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for determining the virtual memory usage rate of a Flink job according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which, when executed by a processor, are used to implement the method for determining the virtual memory usage rate of a Flink job according to any one of claims 1 to 7.

Images