Physical memory management method and device, equipment and medium
By obtaining the physical memory indicators and process identification information of the Flink job, determining the used physical memory and reassigning memory according to the usage rate, the problem of inaccurate usage rate in Flink job memory management is solved, and job stability and performance are improved .
Patent Information
- Application Number
- CN202411389169.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-03
AI Technical Summary
The existing Flink job memory management scheme is inaccurate in determining memory usage, resulting in insufficient stability of Flink jobs.
By obtaining the physical memory indicators and process identification information of the Flink job, determining the physical memory that has been used, and memory redistribution is performed based on the physical memory usage and preset thresholds, ensuring that each Flink job obtains the appropriate physical memory resources.
It realizes accurate determination of the physical memory usage of Flink jobs, improves the stability and performance of jobs, reduces system processing delays, and improves memory resource utilization.
Smart Images
Figure CN119248501A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of memory management, and in particular to a physical memory management method, device, equipment and medium. Background Art
[0002] Flink is an open source stream processing framework, the core of which is a distributed streaming data flow engine, which is used to perform stateful computing on unbounded and bounded data streams. The stable operation of Flink jobs is affected by memory factors.
[0003] In the current solution, memory management of Flink jobs is mainly achieved by collecting Flink's total memory and its usage rate.
[0004] However, existing memory management solutions have the problem of inaccurate determination of memory usage, resulting in insufficient stability of Flink jobs. Summary of the invention
[0005] The present application provides a physical memory management method, apparatus, device and medium to solve the problem that the existing memory management solution has inaccurate determination of memory usage, resulting in insufficient stability of Flink jobs.
[0006] In a first aspect, the present application provides a method for managing physical memory, the method comprising:
[0007] Get the physical memory indicators and process identification information corresponding to the Flink job;
[0008] Determine the process data corresponding to the Flink job based on the process identification information;
[0009] Determine the physical memory used by the Flink job based on the process data;
[0010] Determine the physical memory usage of the Flink job based on the physical memory indicator and the used physical memory;
[0011] Based on the physical memory usage and the preset threshold, the physical memory of the Flink job is reallocated to obtain the corresponding physical memory after reallocation.
[0012] In the embodiment of the present application, obtaining the physical memory indicator and process identification information corresponding to the Flink job includes:
[0013] Determine the program interface and cluster nodes corresponding to the Flink job;
[0014] Collect the physical memory indicators of Flink jobs according to the program interface;
[0015] Determine the process identification information based on the cluster node.
[0016] In the embodiment of the present application, determining the physical memory used by the Flink job according to the process data includes:
[0017] Determine anonymous memory data, file memory data, and shared memory data in process data;
[0018] The used physical memory is determined based on the sum of the anonymous memory data, the file memory data, and the shared memory data.
[0019] In the embodiment of the present application, the physical memory usage of the Flink job is determined according to the physical memory indicator and the used physical memory, including:
[0020] According to the physical memory indicator, determine the number of task managers and target physical memory corresponding to the Flink job;
[0021] Determine the target physical memory for the Flink job based on the number of task managers and the target physical memory;
[0022] Determine the physical memory usage based on the used physical memory and the target physical memory.
[0023] In the embodiment of the present application, the physical memory of the Flink job is reallocated according to the physical memory usage and the preset threshold value, and the corresponding physical memory after reallocation is obtained, including:
[0024] If the physical memory usage is greater than the preset threshold, the free physical memory for the Flink job is determined based on the target physical memory of the Flink job and the used physical memory;
[0025] Determine the preset allocation ratio value corresponding to the used physical memory;
[0026] Determine the corresponding physical memory to be updated according to the preset allocation ratio value and the blank physical memory;
[0027] Reallocate the physical memory of the Flink job based on the physical memory to be updated.
[0028] In the embodiment of the present application, determining the preset allocation ratio value corresponding to the used physical memory includes:
[0029] Determine the preset memory range corresponding to the used physical memory;
[0030] According to the preset memory range, a corresponding preset allocation ratio value is determined.
[0031] In an embodiment of the present application, the method further includes:
[0032] Generate corresponding physical memory prompt information based on the physical memory to be updated and the physical memory used by the Flink job;
[0033] Output physical memory prompt information.
[0034] In a second aspect, the present application provides a physical memory management device, the device comprising:
[0035] The information acquisition module is used to obtain the physical memory indicators and process identification information corresponding to the Flink job;
[0036] The data determination module is used to determine the process data corresponding to the Flink job based on the process identification information;
[0037] The memory determination module is used to determine the physical memory used by the Flink job based on the process data;
[0038] The usage determination module is used to determine the physical memory usage of the Flink job based on the physical memory index and the used physical memory;
[0039] The reallocation module is used to reallocate the physical memory of the Flink job according to the physical memory usage and the preset threshold, and obtain the corresponding physical memory after reallocation.
[0040] In a third aspect, the present application provides a computer device, comprising: a processor, and a memory communicatively connected to the processor;
[0041] Memory stores computer-executable instructions;
[0042] The processor executes the computer-executable instructions stored in the memory to implement the method of the present application.
[0043] In a fourth aspect, the present application provides a computer-readable storage medium, in which program code is stored, and when the program code is executed by a processor, it is used to implement the method of the present application.
[0044] The physical memory management method, apparatus, device and medium provided in the present application obtain the physical memory index and process identification information corresponding to the Flink job; determine the process data corresponding to the Flink job according to the process identification information; determine the physical memory used by the Flink job according to the process data; determine the physical memory usage rate of the Flink job according to the physical memory index and the used physical memory; reallocate the physical memory of the Flink job according to the physical memory usage rate and a preset threshold, and obtain the corresponding physical memory after reallocation.
[0045] In this way, the physical memory used by the Flink job can be determined, thereby realizing the determination of the physical memory usage rate of the Flink job, so that the physical memory resources of the Flink job can be reallocated according to the physical memory usage rate, and appropriate physical memory resources can be allocated to each Flink job, thereby improving the stability of the job and better improving efficiency and performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0047] Figure 1 A flow chart of a physical memory management method provided in an embodiment of the present application;
[0048] Figure 2 A flowchart of another physical memory management method provided in an embodiment of the present application;
[0049] Figure 3 A method diagram of a physical memory management method provided in an embodiment of the present application;
[0050] Figure 4 A schematic diagram of the structure of a physical memory management device provided in an embodiment of the present application;
[0051] Figure 5 This is a structural block diagram of a device for executing a physical memory management method according to an embodiment of the present application.
[0052] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0054] In the prior art, Flink's memory management has some defects and limitations. For example, the management of job performance focuses on the main total memory, but does not involve physical memory, especially the physical memory of the task manager. In addition, the calculation of the physical memory usage of the task manager is not involved. In actual applications, the task manager frequently sends the most stringent SIGKILL signal to terminate the job because the physical memory usage of the taskmanager process of the Flink job exceeds the quota, resulting in job restarts and crashes. Therefore, it is very important to evaluate the physical memory usage of the task manager process of the Flink job, allocate appropriate physical memory resources to each Flink, and enhance the stable operation of the Flink job.
[0055] In order to solve the above problems, an embodiment of the present application provides a physical memory management method, which can calculate the taskmanager physical memory and collect the physical memory usage rate by utilizing the collection method combining the cluster task manager node process used and the Flink REST API; at the same time, Prometheus is used to accurately push the optimized Flink job information to relevant operation and maintenance personnel, thereby reducing the operation and maintenance costs.
[0056] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0057] Figure 1 A flow chart of a physical memory management method provided in an embodiment of the present application. Figure 1 As shown, the physical memory management method may include the following steps:
[0058] S110. Obtain the physical memory indicator and process identification information corresponding to the Flink job.
[0059] Among them, Flink is a framework and distributed processing engine for performing stateful computing on unbounded and bounded data streams; a Flink job is a data processing flow consisting of a series of tasks, each task is composed of one or more operators, and the operator is the core component of data conversion or calculation. The execution of Flink jobs is distributed, and tasks can be executed on multiple computing nodes to achieve high throughput and low latency data processing.
[0060] Physical memory indicators refer to job indicator data related to the determination of physical memory in Flink jobs. They can be used to determine the maximum physical memory allocated to Flink jobs, including task manager physical memory, number of task slots, and other indicator data.
[0061] The process identification information is identification information related to the process of the Flink job. According to the process identification information, the process information of the Flink job can be determined. For example, the process identification information can be the application ID or the node process ID pid of the Flink job, so that the corresponding process information can be further determined according to the process identification information.
[0062] Based on this, the physical memory indicators and process identification information corresponding to the Flink job are determined so that the physical memory and corresponding memory usage of the Flink job can be determined based on the memory indicators and identification information.
[0063] S120. Determine the process data corresponding to the Flink job according to the process identification information.
[0064] The process data is the data of the Flink job process corresponding to the process identification information, which is used to determine the physical memory used by the Flink job process, such as physical memory VmRSS, parent process number, anonymous RSS memory RSSAnon, file RSS memory RssFile, and shared memory RssShmem.
[0065] Based on this, the process data corresponding to the identifier is determined through the process identifier information, so that the physical memory used by the process can be determined based on the process data later.
[0066] S130. Determine the physical memory used by the Flink job according to the process data.
[0067] The used physical memory is the memory occupied by the Flink job process.
[0068] Based on this, by determining the process data, the physical memory occupied by the Flink job process can be determined, so that the physical memory of the Flink job can be reasonably managed and controlled based on the occupied physical memory, thereby improving the stability of the Flink job.
[0069] S140. Determine the physical memory usage of the Flink job according to the physical memory indicator and the used physical memory.
[0070] The physical memory usage rate is the ratio of the physical memory occupied by the current process of the Flink job to the total physical memory, which can represent the usage of physical memory by the current task manager process of the Fink job.
[0071] Based on this, the corresponding physical memory usage rate is determined through the physical memory index and the used physical memory, so that the physical memory reallocation of the Flink job process can be determined based on the physical memory usage rate, thereby ensuring that the physical memory of the process can be used normally and achieving stable operation of the Flink job.
[0072] S150. Reallocate the physical memory of the Flink job according to the physical memory usage and a preset threshold value to obtain the corresponding physical memory after reallocation.
[0073] The preset threshold is a pre-set threshold that is compared with the physical memory usage, so as to determine whether to reallocate the physical memory of the Flink job according to the comparison result.
[0074] Redistribution means reallocating the physical memory of the Flink job based on the used physical memory, so as to ensure that the physical memory of the Flink job can run stably after reallocation.
[0075] Based on this, the reallocation of the physical memory of the Flink job is determined according to the physical memory usage and the preset threshold, so that the reallocated physical memory can achieve the stability of the Flink job operation and avoid the crash and restart of the Flink job due to insufficient physical memory.
[0076] In order to solve the problem that the existing memory management solutions do not accurately determine the memory range of Flink, especially the problem that they do not involve the determination of physical memory and the corresponding memory usage rate, the physical memory indicators and process identification information corresponding to the Flink job can be determined, and then the process data corresponding to the identification can be determined according to the process identification information, and the physical memory used by the Flink job process can be further determined according to the process data, and then the physical memory usage rate of the Flink job can be determined according to the determined used physical memory and the obtained physical memory indicators, so as to clarify the current physical memory usage of the Flink job, and further according to the comparison result of the physical memory usage rate and the preset threshold, the physical memory of the Flink job is reallocated, so that the Flink job can run and work according to the reallocated physical memory.
[0077] In this way, by obtaining the physical memory indicators and process identification information corresponding to the Flink job, the physical memory used by the Flink job process and the physical memory usage rate can be determined, so that the physical memory of the Flink job can be reasonably managed according to the physical memory usage rate, and appropriate physical memory resources can be allocated to each Flink job, thereby improving job stability, better improving efficiency and performance, and at the same time improving the utilization of memory resources and reducing the processing latency of the system, thereby better meeting user needs.
[0078] Based on the feasible implementation of S110 above, the present application further provides a process of determining physical memory indicators and process identification information through a program interface and cluster nodes:
[0079] Determine the program interface and cluster nodes corresponding to the Flink job;
[0080] Collect the physical memory indicators of Flink jobs according to the program interface;
[0081] Determine the process identification information based on the cluster node.
[0082] Among them, the program interface can use Flink Rest API to collect and record job indicators, and collect current job indicators according to http: / / / appId / v1 / jobs / jobId / , for example, they can include start time, status, task name and maximum parallelism of tasks. Secondly, http: / / *** / application_id / v1 / jobmanager / config collects ApaceFlink Job performance indicators, including task manager physical memory and number of task slots.
[0083] The cluster nodes are all nodes on the Linux cluster, that is, the physical memory used by all process task managers of jobs on the Linux server is collected through the script. On the Linux cluster, each cluster node is traversed, and all node process numbers pid of each application ID are obtained according to the command.
[0084] Based on this, physical memory indicators are collected through the program interface, and the process identification information of the Flink job is determined by traversing each cluster node on the Linux cluster, thereby realizing the determination of physical memory indicators and process identification information.
[0085] Based on the feasible implementation of S130 above, the present application further provides a process for determining the used physical memory according to the anonymous memory data, the file memory data, and the shared memory data in the process data:
[0086] Determine anonymous memory data, file memory data, and shared memory data in process data;
[0087] The used physical memory is determined based on the sum of the anonymous memory data, the file memory data, and the shared memory data.
[0088] Among them, anonymous memory data, file memory data, and shared memory data can be determined by reading Linux system files according to each process number pid, and the file path is such as / proc / ${pid} / status; the ${pid} variable is pid, and the collection process uses physical memory VmRSS, parent process number, anonymous RSS memory RSSAnon, file RSS memory RssFile and shared memory RssShmem, that is, Key is the Value value corresponding to pPid, VmRSS, RSSAnon, RssFile, and RssShmem respectively.
[0089] The formula for the physical memory occupied by a process is: VmRSS=RSSAnon+RssFile+RssShmem
[0090] Based on this, the Linux system file is read through the process ID pid to determine the corresponding anonymous memory data, file memory data, and shared memory data, so as to further determine the physical memory occupied by the process in the Flink job, so that the corresponding physical memory usage rate can be determined based on the used physical memory, and the physical memory of the Flink job can be reallocated based on the physical memory usage rate to ensure the stability of the job.
[0091] Based on the feasible implementation of S140 above, the present application further provides a process for determining the physical memory usage by determining the number of task managers and target physical memory corresponding to the Flink job:
[0092] According to the physical memory indicator, determine the number of task managers and target physical memory corresponding to the Flink job;
[0093] Determine the target physical memory for the Flink job based on the number of task managers and the target physical memory;
[0094] Determine the physical memory usage based on the used physical memory and the target physical memory.
[0095] Among them, the task manager is the task manager in the operating system. It is a tool for monitoring and managing computer system resources. It provides information about computer performance and displays information about programs and processes running on the computer. It can display the most commonly used units for measuring process performance.
[0096] The target physical memory can be understood as the maximum usable physical memory allocated by the system, that is, the maximum physical memory of the task manager and Flink job.
[0097] Based on this, by determining the number of task managers in the Flink job and the maximum physical memory allocated to them, the maximum physical memory of all task managers can be determined. That is, the maximum physical memory of the Flink job can be determined based on the physical memory of the task manager, so that the corresponding physical memory usage can be determined based on the physical memory used by the Flink job and the maximum physical memory.
[0098] In this embodiment, in order to solve the problem that the existing memory management solution does not involve the determination of the physical memory usage rate, the task manager physical memory can be calculated and the physical memory usage rate can be collected by utilizing the collection method combined with the cluster task manager node process and the Flink REST API, that is, all Flink jobs to be collected are obtained, the physical memory usage rate of the Flink job taskmanager process is collected and recorded, and the taskmanager physical memory of the job is automatically adjusted dynamically.
[0099] So, according to the system script / proc / <pid> / status method to read the size of physical memory occupied, and combine it with the taskmanager memory usage calculation related to Flink ON YARN. It can allocate appropriate physical memory resources to each Flink job, thereby improving the stability of the job.
[0100] Figure 2 A flow chart of another physical memory management method provided in an embodiment of the present application. Figure 2 As shown, the physical memory management method may include the following steps:
[0101] S210. Obtain the physical memory indicator and process identification information corresponding to the Flink job.
[0102] S220. Determine the process data corresponding to the Flink job according to the process identification information.
[0103] S230. Determine the physical memory used by the Flink job according to the process data.
[0104] S240. Determine the physical memory usage of the Flink job according to the physical memory indicator and the used physical memory.
[0105] In this embodiment, the specific implementation of step S210 to step S240 can refer to the content of the above-mentioned embodiment, which will not be repeated here.
[0106] S250: If the physical memory usage is greater than a preset threshold, determine the free physical memory of the Flink job according to the target physical memory of the Flink job and the used physical memory.
[0107] The target physical memory is the maximum physical memory of the Flink job, and the blank physical memory is the currently unoccupied physical memory that can be used for memory reallocation.
[0108] Based on this, the physical memory usage is compared with the preset threshold to obtain a comparison result. If the comparison result is that the physical memory usage is greater than the preset threshold, the blank physical memory is determined according to the target physical memory and the used physical memory of the Flink job, so that the physical memory of the Flink job can be reallocated according to the blank physical memory.
[0109] S260: Determine a preset allocation ratio value corresponding to the used physical memory.
[0110] The preset allocation ratio value is used to determine the blank physical memory that should be allocated when the physical memory of the Flink job is reallocated, which may generally be 5%. The specific value of the preset allocation ratio value may be determined by the user, and this embodiment does not impose any limitation on this.
[0111] Based on this, by determining the preset allocation ratio value corresponding to the physical memory used by the Flink job, the blank physical memory that the Flink job should be allocated to during redistribution can be determined based on the preset allocation ratio value, thereby determining the new physical memory for the Flink job and ensuring stable operation of the job.
[0112] S270: Determine the corresponding physical memory to be updated according to the preset allocation ratio value and the blank physical memory.
[0113] The physical memory to be updated is the blank physical memory added to the Flink job after reallocation. For example, the preset allocation ratio value corresponding to the current Flink job is 5%, and the blank physical memory is 100. Then the physical memory to be updated corresponding to the current Flink job is 5, that is, 5 is added to the physical memory of the Flink job after reallocation. In this way, the new physical memory of the Flink job is determined according to the physical memory to be updated to ensure the stable operation of the Flink job.
[0114] Based on this, the corresponding physical memory to be updated is determined through the preset allocation ratio value corresponding to the Flink job and the blank physical memory of the current system, so that the physical memory of the Flink job can be updated according to the physical memory to be updated later, that is, the physical memory of the Flink job is increased, thereby ensuring the stability of the Flink job operation.
[0115] S280. Reallocate the physical memory of the Flink job according to the physical memory to be updated.
[0116] Based on this, the physical memory to be updated corresponding to the Flink job is determined, and then the physical memory of the Flink job is increased according to the physical memory.
[0117] Based on the feasible implementation of S260 above, the present application further provides a process for determining a corresponding preset allocation ratio value according to a preset memory range:
[0118] Determine the preset memory range corresponding to the used physical memory;
[0119] According to the preset memory range, a corresponding preset allocation ratio value is determined.
[0120] Among them, the preset memory range is a pre-set memory value range used to determine the corresponding allocation ratio value according to the size of the memory, that is, by determining the memory size range corresponding to the used physical memory, the allocation ratio value corresponding to the size range is determined, that is, the local ratio value corresponding to the used physical memory; for example, the used physical memory is 100, and the corresponding preset memory range is 95-105, and the ratio value corresponding to the used physical memory is 5% of the allocation ratio value corresponding to the memory range of 95-105.
[0121] Based on the above-mentioned S280 feasible implementation method, the present application further provides a process of generating and outputting physical memory prompt information:
[0122] Generate corresponding physical memory prompt information based on the physical memory to be updated and the physical memory used by the Flink job;
[0123] Output physical memory prompt information.
[0124] Among them, the physical memory prompt information is used to prompt relevant personnel of the real-time status of the physical memory and physical memory usage of Flink jobs, so that relevant personnel can better manage and control the physical memory. For example, since Flink jobs are all managed by Stream Park, the dynamic tuning of Apache Flink jobs must be based on Stream Park. Use the remote call service provided by the StreamPark platform to take out the jobs to be tuned in the queue, modify the taskmanager physical memory configuration parameters of the job according to the above rules, and restart the job. Finally, select the Prometheus monitoring and alarm solution, collect monitoring performance indicators, manage indicator data, and then provide alarm services and accurately push standardized alarm information. After completing the dynamic optimization of the job, the list of the optimized jobs and their indicator information "job id", "job name", "job responsible person", "job start time", "job team", "taskmanager physical memory usage", and "remarks" are exposed to the Prometheus Server through metrics data. Prometheus uses the active pull mode to collect monitoring indicators and uses its own time series database to store monitoring data. Alertmanager is selected as the alarm management component, alarm routing is set, and email notification method is selected. At the same time, alarm rules are defined in Prometheus.
[0125] Based on this, by determining the physical memory of the Flink job, physical memory usage and other information related to the physical memory, the corresponding physical memory prompt information is generated, and the prompt information can be sent through the corresponding platform such as the Stream Park platform, so as to realize the monitoring and alarm of the physical memory, so that the relevant management personnel can control it.
[0126] Figure 3 A method diagram of a physical memory management method provided in an embodiment of the present application is shown as follows: Figure 3 First, the physical memory indicators collected and recorded are divided into two parts. One part is fixed performance indicators, that is, indicators that are basically fixed after the task is started. The other part is dynamic performance indicators, that is, performance indicators that will change dynamically in real time after the task is started. Fixed indicators do not need to be monitored in real time. They only need to be collected and recorded in mysql-1 after daily inspection or task restart. The recording method is to overwrite historical records. Dynamic indicators are collected and recorded in mysql-2 and mysql-3 regularly every 5 minutes at a certain frequency. The recording method is incremental recording, and the indicator records for the past week are saved. The method for collecting indicators is Apache Flink Rest API and Linux system files. The API address is: http: / / *** / appId / v1 / jobs / jobId / , and the system file path is: / proc / ${pid} / status; Secondly, using the architecture and memory management model of Apache Flink, the physical memory usage of the taskmanager process is calculated based on the recorded physical memory indicators. Then, based on the rules, the taskmanager.memory.managed.fraction parameter of the Fli job is automatically and dynamically adjusted, the job is restarted, and the physical memory resource optimization of the job is completed; finally, the Prometheus monitoring and alarm solution is selected to provide alarm services and accurately push standardized alarm information.
[0127] In this embodiment, if the physical memory usage rate is greater than a preset threshold, the preset memory range corresponding to the physical memory used by the Flink job and the allocation ratio value corresponding to the memory range are determined, so that the physical memory to be updated is determined according to the allocation ratio value and the blank physical memory, thereby realizing the reallocation of the physical memory; at the same time, in combination with stream park, the dynamic adjustment of the physical memory configuration of the task manager process and the restart of the job are realized.
[0128] In this way, it supports automatic adjustment of physical memory indicators to restart jobs, and further monitors and alerts jobs that have completed optimization. Prometheus is used to accurately push optimized job information to relevant operation and maintenance personnel, reducing operation and maintenance costs.
[0129] Figure 4 Schematic diagram of a physical memory management device 400 provided in an embodiment of the present application. Figure 4 As shown, the physical memory management device 400 includes: an information acquisition module 410 , a data determination module 420 , a memory determination module 430 , a usage rate determination module 440 , and a reallocation module 450 .
[0130] The information acquisition module 410 is used to obtain the physical memory indicators and process identification information corresponding to the Flink job;
[0131] The data determination module 420 is used to determine the process data corresponding to the Flink job according to the process identification information;
[0132] A memory determination module 430 is used to determine the physical memory used by the Flink job according to the process data;
[0133] A usage determination module 440 is used to determine the physical memory usage of the Flink job based on the physical memory indicator and the used physical memory;
[0134] The reallocation module 450 is used to reallocate the physical memory of the Flink job according to the physical memory usage and a preset threshold value to obtain the corresponding physical memory after reallocation.
[0135] In the embodiment of the present application, the information acquisition module 410 may also be specifically used for:
[0136] Determine the program interface and cluster nodes corresponding to the Flink job;
[0137] Collect the physical memory indicators of Flink jobs according to the program interface;
[0138] Determine the process identification information based on the cluster node.
[0139] In the embodiment of the present application, the memory determination module 430 may also be specifically used for:
[0140] Determine anonymous memory data, file memory data, and shared memory data in process data;
[0141] The used physical memory is determined based on the sum of the anonymous memory data, the file memory data, and the shared memory data.
[0142] In the embodiment of the present application, the usage rate determination module 440 may also be specifically used for:
[0143] According to the physical memory indicator, determine the number of task managers and target physical memory corresponding to the Flink job;
[0144] Determine the target physical memory for the Flink job based on the number of task managers and the target physical memory;
[0145] Determine the physical memory usage based on the used physical memory and the target physical memory.
[0146] In the embodiment of the present application, the redistribution module 450 may also be specifically used for:
[0147] If the physical memory usage is greater than the preset threshold, the free physical memory for the Flink job is determined based on the target physical memory of the Flink job and the used physical memory;
[0148] Determine the preset allocation ratio value corresponding to the used physical memory;
[0149] Determine the corresponding physical memory to be updated according to the preset allocation ratio value and the blank physical memory;
[0150] Reallocate the physical memory of the Flink job based on the physical memory to be updated.
[0151] In the embodiment of the present application, the redistribution module 450 may also be specifically used for:
[0152] Determine the preset memory range corresponding to the used physical memory;
[0153] According to the preset memory range, a corresponding preset allocation ratio value is determined.
[0154] In the embodiment of the present application, the modules included in the physical memory management device 400 may also be specifically used for:
[0155] Generate corresponding physical memory prompt information based on the physical memory to be updated and the physical memory used by the Flink job;
[0156] Output physical memory prompt information.
[0157] The physical memory management device 400 provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.
[0158] Figure 5 This is a schematic diagram of the structure of the device provided in the embodiment of the present application. Figure 5 As shown, the device 500 includes:
[0159] The device 500 may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a communication component 503 and other components. The processor 501 , the memory 502 and the communication component 503 are connected via a bus 504 .
[0160] In a specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, so that at least one processor 501 executes the above physical memory management method.
[0161] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0162] In the above Figure 5 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0163] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.
[0164] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.
[0165] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the steps in any of the above-mentioned physical memory management methods when executed by a processor.
[0166] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0167] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0168] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple program codes are stored. The program codes can be loaded by a processor to execute the steps in any physical memory management method provided in the embodiment of the present application.
[0169] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0170] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.
[0171] Since the instructions stored in the storage medium can execute the steps in any physical memory management method provided in the embodiments of the present application, the beneficial effects that can be achieved by any physical memory management method provided in the embodiments of the present application can be achieved. Please see the previous embodiments for details and will not be repeated here.
[0172] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0173] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.< / pid>
Claims
1. A method for managing physical memory, characterized in that: include: Get the physical memory indicators and process identification information corresponding to the Flink job; Determine the process data corresponding to the Flink job according to the process identification information; Determine the physical memory used by the Flink job according to the process data; Determine the physical memory usage of the Flink job according to the physical memory indicator and the used physical memory; The physical memory of the Flink job is reallocated according to the physical memory usage and a preset threshold value to obtain the physical memory corresponding to the reallocated memory.
2. The method according to claim 1, characterized in that The method of obtaining the physical memory indicators and process identification information corresponding to the Flink job includes: Determine the program interface and cluster nodes corresponding to the Flink job; According to the program interface, collecting the physical memory indicator of the Flink job; The process identification information is determined according to the cluster node.
3. The method according to claim 1 or 2, characterized in that: Determining the physical memory used by the Flink job according to the process data includes: Determine anonymous memory data, file memory data, and shared memory data in the process data; The used physical memory is determined according to the sum of the anonymous memory data, the file memory data, and the shared memory data.
4. The method according to claim 1 or 2, characterized in that: The determining the physical memory usage rate of the Flink job according to the physical memory indicator and the used physical memory includes: According to the physical memory indicator, determine the number of task managers and target physical memory corresponding to the Flink job; Determine the target physical memory of the Flink job according to the number of the task managers and the target physical memory; The physical memory usage rate is determined according to the used physical memory and the target physical memory.
5. The method according to claim 1 or 2, characterized in that: The reallocating the physical memory of the Flink job according to the physical memory usage rate and the preset threshold value to obtain the physical memory corresponding to the reallocated memory includes: If the physical memory usage rate is greater than the preset threshold, determining the blank physical memory of the Flink job according to the target physical memory of the Flink job and the used physical memory; Determine a preset allocation ratio value corresponding to the used physical memory; Determining the corresponding physical memory to be updated according to the preset allocation ratio value and the blank physical memory; The physical memory of the Flink job is reallocated according to the physical memory to be updated.
6. The method according to claim 5, characterized in that The determining of the preset allocation ratio value corresponding to the used physical memory includes: Determine a preset memory range corresponding to the used physical memory; According to the preset memory range, the corresponding preset allocation ratio value is determined.
7. The method according to claim 1 or 2, characterized in that: The method further comprises: Generate corresponding physical memory prompt information according to the physical memory to be updated of the Flink job and the used physical memory; Output the physical memory prompt information.
8. A physical memory management device, characterized in that: The device comprises: The information acquisition module is used to obtain the physical memory indicators and process identification information corresponding to the Flink job; A data determination module, used to determine the process data corresponding to the Flink job according to the process identification information; A memory determination module, used to determine the physical memory used by the Flink job according to the process data; A usage rate determination module, used to determine the physical memory usage rate of the Flink job according to the physical memory indicator and the used physical memory; The reallocation module is used to reallocate the physical memory of the Flink job according to the physical memory usage rate and a preset threshold value to obtain the physical memory corresponding to the reallocated memory.
9. A device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, the one or more programs being configured to execute the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.