Spark memory utilization rate determination method and device, equipment and storage medium

Abstract

The invention provides a Spark memory utilization rate determination method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: firstly, obtaining an application program identifier corresponding to a Spark job; then, according to the application program identifier, determining a node process identifier corresponding to the Spark job in each cluster node, each cluster node being a running node of the Spark job; according to the node process identifier, based on the system file path, a used physical memory corresponding to the node process stored in the system file is read; and finally, determining the memory utilization rate of the Spark operation according to the used physical memory and the total allocated physical memory of the Spark operation. According to the method and the device, more accurate physical memory use data can be obtained, so that more accurate memory utilization rate of the Spark is obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for determining Spark memory utilization. Background Art

[0002] Spark is a general engine that can be used to perform a variety of operations, including Structured Query Language (SQL) queries, text processing, machine learning, etc. It also supports multiple resource managers, such as standalone, Standalone, Hadoop YARN, Apache Mesos, etc. Users can flexibly choose the operation mode according to the existing big data platform. Spark runs in memory, so memory management is a very important aspect of Spark, and memory needs to be managed effectively to avoid performance problems and crashes.

[0003] In the prior art, the performance monitoring of Spark jobs usually focuses on the performance monitoring items, while the physical memory is generally monitored in real time through the Spark User Interface (Spark UI) to view the execution status of the job, so as to understand the memory consumption of each task.

[0004] However, the above methods can only provide a summary of memory usage, resulting in insufficient accuracy in determining the memory utilization of Spark jobs. Summary of the invention

[0005] The present application provides a Spark memory utilization determination method, apparatus, device and storage medium to solve the technical problem of insufficient accuracy in determining the memory utilization of Spark jobs in the prior art.

[0006] In a first aspect, the present application provides a method for determining Spark memory utilization, comprising:

[0007] Get the application ID corresponding to the Spark job;

[0008] According to the application identifier, the node process identifier corresponding to the Spark job is determined in each cluster node, and each cluster node is the running node of the Spark job;

[0009] According to the node process identifier and the system file path, read the used physical memory corresponding to the node process stored in the system file;

[0010] Determine the memory utilization of the Spark job based on the used physical memory and the total physical memory allocated to the Spark job.

[0011] Optionally, according to the node process identifier and based on the system file path, reading the used physical memory corresponding to the node process stored in the system file includes:

[0012] According to the node process identifier, determine the anonymous resident set size memory, the file resident set size memory and the shared resident set size memory of each process;

[0013] Determines the used physical memory of the node process based on the anonymous resident set size memory, file resident set size memory, and shared resident set size memory of each process.

[0014] Optionally, after determining the used physical memory of the node process, the method further includes:

[0015] Determine the information related to the use of physical memory of the node process, the information related to the use of physical memory includes the node process identifier, the parent node process identifier, the anonymous resident set size process, the file resident set size process, the shared resident set size process, and the used physical memory of the node process; store the information related to the use of physical memory of the node process in the first transactional relationship data table;

[0016] And / or, determining information related to allocated physical memory of the Spark job, where the information related to allocated physical memory includes an application identifier, a maximum degree of parallelism, and physical memory of each working node in the Spark job; and storing the information related to allocated physical memory of the Spark job in a second transactional relational data table.

[0017] Optionally, the method further comprises:

[0018] The data in the first transactional relationship data table and / or the data in the second transactional relationship data table are periodically synchronized to the parallel analysis database, and a first parallel analysis data table and a second parallel analysis data table are correspondingly generated to form report data, which is used for process tracking and data analysis;

[0019] Correspondingly, the synchronized data in the first transactional relationship data table and the second transactional relationship data table are deleted.

[0020] Optionally, the parallel analysis database sets hot and cold data stratification, and the method further includes:

[0021] According to the access frequency of the data in the first parallel analysis data table and the second parallel analysis data table, the data is divided into cold data and hot data;

[0022] Move cold data to remote object storage.

[0023] Optionally, determine information about the physical memory allocated to the Spark job, including:

[0024] According to the application program identifier, determining the application program interface corresponding to the application program identifier;

[0025] According to the application program interface and the preset second file path, the information related to the allocated physical memory of the Spark job is determined.

[0026] Optionally, the memory utilization of the Spark job is determined according to the used physical memory and the total physical memory allocated to the Spark job, including:

[0027] Based on the used physical memory, determine the total physical memory used by the Spark job;

[0028] Determine the total physical memory allocated for the Spark job based on the physical memory of each worker node in the Spark job and the number of worker nodes in the Spark job;

[0029] Determine the memory utilization of the Spark job based on the total physical memory used by the Spark job and the total physical memory allocated to the Spark job.

[0030] In a second aspect, the present application provides a Spark memory utilization determination device, comprising:

[0031] The acquisition module is used to obtain the application ID corresponding to the Spark job;

[0032] The first determination module is used to determine the node process identifier corresponding to the Spark job in each cluster node according to the application identifier, and each cluster node is a running node of the Spark job;

[0033] A reading module is used to read the used physical memory corresponding to the node process stored in the system file according to the node process identifier and based on the system file path;

[0034] The second determining module is used to determine the memory utilization of the Spark job according to the used physical memory and the total physical memory allocated to the Spark job.

[0035] Optionally, the reading module is specifically used for:

[0036] According to the node process identifier, determine the anonymous resident set size memory, the file resident set size memory and the shared resident set size memory of each process;

[0037] Determines the used physical memory of the node process based on the anonymous resident set size memory, file resident set size memory, and shared resident set size memory of each process.

[0038] Optionally, after determining the used physical memory of the node process, the reading module is further used to:

[0039] Determine the information related to the use of physical memory of the node process, the information related to the use of physical memory includes the node process identifier, the parent node process identifier, the anonymous resident set size process, the file resident set size process, the shared resident set size process, and the used physical memory of the node process; store the information related to the use of physical memory of the node process in the first transactional relationship data table;

[0040] And / or, determining information related to allocated physical memory of the Spark job, where the information related to allocated physical memory includes an application identifier, a maximum degree of parallelism, and physical memory of each working node in the Spark job; and storing the information related to allocated physical memory of the Spark job in a second transactional relational data table.

[0041] Optionally, the reading module is further configured to:

[0042] The data in the first transactional relationship data table and / or the data in the second transactional relationship data table are periodically synchronized to the parallel analysis database, and a first parallel analysis data table and a second parallel analysis data table are correspondingly generated to form report data, which is used for process tracking and data analysis;

[0043] Correspondingly, the synchronized data in the first transactional relationship data table and the second transactional relationship data table are deleted.

[0044] Optionally, the parallel analysis database sets up hot and cold data tiering, and the read module is also used to:

[0045] According to the access frequency of the data in the first parallel analysis data table and the second parallel analysis data table, the data is divided into cold data and hot data;

[0046] Move cold data to remote object storage.

[0047] Optionally, determine information related to the physical memory allocated for the Spark job. The read module is also used to:

[0048] According to the application identifier, determine the application interface corresponding to the application identifier;

[0049] According to the application program interface and the preset second file path, the information related to the allocated physical memory of the Spark job is determined.

[0050] Optionally, the second determining module is specifically configured to:

[0051] Based on the used physical memory, determine the total physical memory used by the Spark job;

[0052] Determine the total physical memory allocated for the Spark job based on the physical memory of each worker node in the Spark job and the number of worker nodes in the Spark job;

[0053] Determine the memory utilization of the Spark job based on the total physical memory used by the Spark job and the total physical memory allocated to the Spark job.

[0054] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0055] Memory stores computer-executable instructions;

[0056] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect and various possible implementations of the first aspect.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the method of the above-mentioned first aspect and various possible implementation methods of the first aspect is executed.

[0058] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method as described in the first aspect and various possible implementation methods of the first aspect.

[0059] The Spark memory utilization determination method, apparatus, device and storage medium provided in the embodiments of the present application can accurately monitor the memory usage of each Spark job on each cluster node by obtaining the application identifier of the Spark job and determining the node process identifier corresponding to the Spark job in each cluster node, and then directly read the physical memory usage of the node process stored in the system file based on the node process identifier and the system file path to obtain more accurate physical memory consumption data. Therefore, by reading the used physical memory corresponding to the node process stored in the system file and combining the total physical memory allocated to the Spark job, the memory utilization of the Spark job can be calculated in real time. This method can provide accurate memory usage data and comprehensively cover the memory usage of the entire Spark job, rather than just the data of a certain node or part of the nodes, thereby obtaining more accurate memory utilization information of Spark. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 1 ;

[0062] Figure 2 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 2 ;

[0063] Figure 3 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 3 ;

[0064] Figure 4 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 4 ;

[0065] Figure 5 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 5 ;

[0066] Figure 6 A schematic diagram of the structure of a Spark memory utilization determination device provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0068] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0069] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0070] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0071] First, the terms involved in this application are explained:

[0072] Apache Spark: It is a fast and general computing engine designed for large-scale data processing. Spark's cluster architecture mainly consists of five parts: cluster manager, worker, executor, driver, and application.

[0073] Cluster Manager is the Spark cluster manager, which is mainly used to manage and allocate the resources of the entire cluster. According to different deployment modes, it can be divided into different types of managers such as Local, Standalone, YARN, Mesos, AWS, etc.

[0074] Worker is a Spark working node used to execute submitted tasks.

[0075] Executor is the component that actually executes computing tasks. It is a process of an Application running on a Worker.

[0076] Driver is the driver of Application.

[0077] Application is an application written based on the Spark Application Programming Interface (API).

[0078] In the prior art, when a Spark job is running, the Spark UI provides a network interface on a specific port, where a list of all jobs can be viewed. Each job entry usually includes information such as the job identifier (ID), description, status, submission time, and running time. On different interfaces of the Spark UI, detailed information of the job can be queried, including the memory usage of each task and the memory usage of each executor.

[0079] However, the data in the Spark UI comes from statistics and summaries at the application layer, and usually displays high-level memory usage information, lacking detailed memory usage for each process. The memory usage information is usually generalized and cannot reflect real-time memory usage. Data refresh and update may be delayed, making it impossible to accurately calculate the memory utilization of Spark jobs.

[0080] In the process of solving the above technical problems, the inventors' technical ideas are as follows: Since the data of Spark UI may introduce errors or inaccuracies due to the increase of abstraction levels, if the memory usage data provided by the operating system is directly read to obtain the actual physical memory consumption of a single Spark process on each node, the abstraction of the application layer can be bypassed to provide more accurate memory usage data. On this basis, if the query is performed through the application ID and the node process ID, the memory usage of each process can be accurately located in real time, which can not only collect more fine-grained physical memory, but also ensure the real-time nature of memory usage information, thereby gaining an in-depth understanding of the memory consumption of Spark jobs and improving the accuracy of determining the memory utilization of Spark jobs.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] Figure 1 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 1 .like Figure 1 As shown, the method may include the following steps:

[0083] S101. Obtain an application program identifier corresponding to a Spark job.

[0084] The application ID may refer to a unique ID assigned to each Spark job when it is submitted, namely, applicationId, which is used to identify the job in the cluster. This ID remains unchanged throughout the life cycle of the job.

[0085] You can use a stream processing application development and management framework (such as Stream Park) to obtain all Spark jobs that need to be monitored on an operating system (such as Linux). Stream Park can list the currently running Spark jobs and provide the applicationId of each job. At the same time, Stream Park can also serve as a monitoring system to support and monitor the entire life cycle of stream processing jobs from development to launch.

[0086] Optionally, in addition to the application ID, Stream Park can also obtain the job name (job ID) of each job to help better manage and record job information. The job name is usually specified by the user when submitting a job to facilitate identification of the purpose or source of the job.

[0087] In this embodiment, the operating environment is dynamic, jobs are continuously submitted and completed, and the monitoring system automatically obtains and updates the application identifier to always track the currently active jobs.

[0088] It can be understood that by obtaining the application identifier corresponding to the Spark job, the status and resource usage of each Spark job can be accurately tracked, laying the foundation for subsequent memory utilization monitoring and resource management, ensuring the accuracy of monitoring and the efficiency of management.

[0089] S102: According to the application program identifier, a node process identifier corresponding to the Spark job is determined in each cluster node, and each cluster node is a running node of the Spark job.

[0090] The node process identifier may refer to an identifier (Process ID, PID) of a process running in the operating system. In this embodiment, for a Spark job, each Executor will correspond to one or more processes when running on a cluster node, and the PIDs of these processes are the node process identifiers. The node process identifiers related to a specific Spark job can be screened out by using system tools or other monitoring tools on the cluster nodes in combination with the application identifier.

[0091] Exemplarily, a command line tool, such as a process status (ps) command or a script, may be used to search for a process command line parameter containing a specific application identifier, thereby determining a corresponding node process identifier.

[0092] Cluster nodes can refer to individual computers or servers that make up a computing cluster. In a Spark cluster, nodes usually include Driver nodes and Executor nodes. These nodes can be identified and managed through cluster management tools (such as YARN, Mesos, or Kubernetes) or Spark's own cluster management capabilities.

[0093] It can be understood that determining the processes associated with a specific job on all cluster nodes can ensure comprehensive monitoring of resource usage of the entire job, not just a single node. On the other hand, in a dynamic environment, jobs may start and stop on different nodes. By determining the node process identity in real time, it can be ensured that the monitoring system can always track the currently active processes.

[0094] S103: Read the used physical memory corresponding to the node process stored in the system file according to the node process identifier and based on the system file path.

[0095] The system file path may refer to a file or directory provided by the operating system for storing and accessing process status and resource usage information. In Linux systems, this is usually the / proc file system. For each process, there is a subdirectory named after the node process identifier (PID) under the / proc directory (such as / proc / ${pid}). This directory contains multiple files, among which the status file provides detailed memory usage information.

[0096] The used physical memory corresponding to the node process may refer to the amount of memory currently occupied by the process in the physical memory, including the Resident Set Size (RSS).

[0097] In this embodiment, the value (Value) corresponding to the field (Key) can be extracted by reading the VmRSS (Virtual Memory Resident Set Size) field, RSSAnon (Anonymous RSS) field, RssFile field, and RssShmem (Shared Memory RSS) field in the / proc / ${pid} / status file, thereby obtaining the physical memory usage of the process. Among them, VmRSS represents the physical memory actually used by the process, in KB; RSSAnon represents the RSS of the anonymous memory (i.e., memory not mapped to a file) used by the process; RssFile represents the RSS of the file-mapped memory used by the process; and RssShmem represents the RSS of the shared memory used by the process.

[0098] Optionally, you can also read the parent process ID (pPid), which is the process that started the process. By obtaining pPid, you can understand the hierarchy and parent-child relationship of the process, which helps with process management and troubleshooting.

[0099] In some embodiments, the subdirectory named after the node process identifier also includes an smaps file, and the physical memory usage of the process can be obtained by reading the RSS information in the / proc / ${pid} / smaps file.

[0100] It is understandable that by directly reading the memory usage data at the operating system level, the actual physical memory consumption of each process can be obtained, which can provide more accurate memory usage information than application-level tools (such as Spark UI). Reading the memory usage data in the system file can reflect the current memory usage in real time and support rapid response to changes in memory usage.

[0101] S104: Determine the memory utilization of the Spark job according to the used physical memory and the total physical memory allocated to the Spark job.

[0102] The total physical memory allocated to a Spark job refers to the total amount of memory allocated to each Executor and Driver of the job when the Spark job is started. It is usually specified in the job configuration, including the memory of each Executor (spark.executor.memory) and the memory of the Driver (spark.driver.memory). You can obtain this by viewing the Spark job configuration file or the parameters when submitting the job.

[0103] The memory utilization of a Spark job can be calculated by calculating the percentage of the sum of the used physical memory corresponding to all node processes of the Spark job in the system file to the total physical memory allocated to the Spark job. By calculating the memory utilization of a Spark job, you can evaluate the efficiency of the Spark job in using the allocated memory resources, which helps to identify whether there is a waste or shortage of memory resources.

[0104] Specifically, if the memory utilization is too low, it may mean that too much memory is allocated, and you can consider reducing the allocation to save resources; if the memory utilization is too high, you may need to increase the memory allocation or optimize the job to avoid performance issues caused by insufficient memory.

[0105] It is understandable that by real-time monitoring of the memory utilization of Spark jobs, the Stream Park platform can support dynamic resource management, help Spark jobs adjust memory allocation strategies at runtime to adapt to changes in job requirements, reduce job failures or system instability caused by insufficient memory, and improve the overall stability and performance of Spark jobs and clusters.

[0106] The Spark memory utilization determination method provided in the embodiment of the present application can read the memory usage of the process on each node by accessing the memory usage information at the operating system level according to the application identifier and the node process identifier corresponding to the Spark job, thereby obtaining more accurate physical memory consumption data and reflecting the current memory usage in real time. Therefore, a more accurate memory utilization of the Spark job can be calculated based on the physical memory consumption data and the total physical memory allocated to the Spark job.

[0107] Based on the above embodiments, Figure 2 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 2 .like Figure 2 As shown, in this embodiment Figure 1Based on the embodiment, S103 is described in detail, which specifically includes the following steps:

[0108] S201. Determine the anonymous resident set size memory, the file resident set size memory, and the shared resident set size memory of each process according to the node process identifier.

[0109] Among them, anonymous resident set size memory (RSSAnon) represents the memory portion used by the process that is not mapped to any file. This part of memory is usually used for heap, stack and other dynamically allocated memory areas. Through anonymous resident set size memory, the dynamic memory requirements of the process can be analyzed and memory allocation can be optimized.

[0110] The file resident set size memory (RssFile) represents the memory portion mapped to the file used by the process, which usually includes memory-mapped files (such as shared libraries) and other file I / O operations. The file resident set size memory can be used to understand the file I / O behavior of the process and optimize the file system performance.

[0111] Shared resident set size memory (RssShmem) represents the shared memory portion used by the process. This portion of memory can be shared by multiple processes and is often used for inter-process communication. By sharing the resident set size memory, inter-process communication can be optimized and memory redundancy can be reduced.

[0112] As an example, you can use system commands to obtain the node process ID of each process and access system files. The system file path is as follows: Figure 1 Embodiment, this embodiment is not repeated here, the file provides detailed memory mapping information, including RSSAnon, RssFile and RssShmem, etc., and then use the script or command line tool to extract the values ​​of RSSAnon, RssFile and RssShmem from the system file to indicate the size of each memory part.

[0113] It can be understood that by identifying the different parts of memory usage (anonymous, file, shared) the resident set size memory helps in more fine-grained performance tuning and resource management.

[0114] S202: Determine the used physical memory of the node process according to the anonymous resident set size memory, the file resident set size memory, and the shared resident set size memory of each process.

[0115] Among them, the used physical memory (VmRSS) of the node process is the sum of the anonymous resident set size memory, file resident set size memory and shared resident set size memory of each process, that is, the information related to the used physical memory of the above node process satisfies the following relationship: VmRSS=RSSAnon+RssFile+RssShmem.

[0116] The Spark memory utilization determination method provided in the embodiment of the present application can accurately calculate the physical memory consumption of the process by collecting fine-grained physical memory, and can better optimize memory allocation and usage efficiency and prevent potential memory bottleneck problems by identifying different parts of memory usage, including anonymous resident set size memory, file resident set size memory, and shared resident set size memory.

[0117] Based on the above embodiments, Figure 3 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 3 .like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, after determining the used physical memory of the node process, the method further includes the following steps:

[0118] S301. Determine information related to the use of physical memory of a node process, where the information related to the use of physical memory includes a node process ID, a parent node process ID, an anonymous resident set size process, a file resident set size process, a shared resident set size process, and physical memory used by the node process; store the information related to the use of physical memory of the node process into a first transactional relationship data table.

[0119] Among them, the node process identifier is the process number of a process in the node, the parent node process identifier can be the parent process number of the process, and the first transactional relationship data table can refer to a database table used to store and manage information related to the use of physical memory by the node process. It supports transactional (i.e., atomicity, consistency, isolation, and persistence) operations and can ensure data consistency and integrity.

[0120] In this embodiment, a relational database management system such as MySQL, PostgreSQL, Oracle, etc. may be used for implementation. These relational databases all support structured queries and complex data operations.

[0121] Optionally, the information related to the use of physical memory of the node process also includes an auto-increment flag, the type of application or command, an interface machine Internet Protocol (IP) address, data entry time, description, reserved fields, collection time, etc.

[0122] As an example, the information related to the physical memory used by the node process is stored in the database MySQL table 1. The structure of table 1 is as follows:

[0123] CREATE TABLE `Table1` (

[0124] `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Auto-increment id',

[0125] `name` varchar(200) NOT NULL COMMENT 'Type of application or command',

[0126] `ip` varchar(200) DEFAULT NULL COMMENT 'Interface machine IP address',

[0127] `applicationId` varchar(200) NOT NULL COMMENT 'job id',

[0128] `containerid` varchar(200) DEFAULT NULL COMMENT 'containerid',

[0129] `pid` bigint(20) DEFAULT NULL COMMENT 'Process number',

[0130] `ppid` bigint(20) DEFAULT NULL COMMENT 'Parent process ID',

[0131] `VmRSS` varchar(200) DEFAULT NULL COMMENT 'Physical memory',

[0132] `RssAnon` varchar(200) DEFAULT NULL COMMENT 'Anonymous RSS memory',

[0133] `RssFile` varchar(200) DEFAULT NULL COMMENT 'File RSS memory',

[0134] `RssShmem` varchar(200) DEFAULT NULL COMMENT 'Shared memory',

[0135] `insertTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Data storage time',

[0136] `desc` varchar(255) DEFAULT NULL COMMENT 'Description',

[0137] `other` varchar(255) DEFAULT NULL COMMENT 'Reserved field',

[0138] `time` varchar(100) DEFAULT NULL COMMENT 'Collection time',

[0139] PRIMARY KEY (`id`)

[0140] ) ENGINE=InnoDB AUTO_INCREMENT=01 DEFAULT CHARSET=utf8mb4;

[0141] In some embodiments, information related to the use of physical memory by a node process may also include central processing unit (CPU) usage, process startup time, user ID, process status, etc. This data information may further help identify and analyze the resource usage profile of the process, thereby determining the current activity status and health of the process.

[0142] S302, and / or, determining information related to allocated physical memory of the Spark job, where the information related to allocated physical memory includes an application identifier, a maximum degree of parallelism, and physical memory of each working node in the Spark job; and storing information related to allocated physical memory of the Spark job in a second transactional relational data table.

[0143] The second transactional relationship data table is similar to the first transactional relationship data table, and may refer to a database table used to store and manage information related to allocated physical memory of Spark jobs.

[0144] Optionally, the information related to the physical memory allocated to the Spark job also includes description, reserved fields, data storage time, etc.

[0145] As another example, the information related to the physical memory allocation of the Spark job is stored in the database MySQL table 2. The structure of table 2 is as follows:

[0146] CREATE TABLE `Table2` (

[0147] `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Auto-increment id',

[0148] `applicationId` varchar(100) NOT NULL COMMENT 'job id',

[0149] `parallelism` varchar(20) NOT NULL COMMENT 'Maximum degree of parallelism',

[0150] `workmanagerSizes` varchar(20) NOT NULL COMMENT 'workmanager physical memory size',

[0151] `desc` varchar(255) DEFAULT NULL COMMENT 'Description',

[0152] `other` varchar(255) DEFAULT NULL COMMENT 'Reserved field',

[0153] `name` varchar(200) NOT NULL COMMENT 'Job name',

[0154] `insertTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Data entry time'

[0155] PRIMARY KEY (`id`)

[0156] ) ENGINE=InnoDB AUTO_INCREMENT=01 DEFAULT CHARSET=utf8mb4;

[0157] In some embodiments, information related to the allocated physical memory of a Spark job may also include Executor memory configuration, Driver memory configuration, job submission time, resource scheduling strategy, job priority, etc. This data information can be used to analyze the execution time and scheduling efficiency of the job, ensure that key jobs are executed first, and thus optimize job performance.

[0158] It is understandable that by storing the information related to the physical memory used by the node process and / or the physical memory allocated by the Spark job in the relational database, the data persistence and manageability can be ensured, and then through transactional operations, the integrity and consistency of the data can be guaranteed to prevent data loss or damage. And through the query capability of the relational database, the physical memory related information of the Spark job can be efficiently retrieved and analyzed, which is convenient for real-time monitoring and historical analysis.

[0159] In one achievable manner, the method further comprises the following steps:

[0160] The data in the first transactional relationship data table and / or the data in the second transactional relationship data table are periodically synchronized to the parallel analysis database, and the first parallel analysis data table and the second parallel analysis data table are correspondingly generated to form report data, which is used for process tracking and data analysis; then the synchronized data in the first transactional relationship data table and the second transactional relationship data table are correspondingly deleted.

[0161] The parallel analysis database can refer to the Doris database, which is used for real-time analysis and reporting. In the Doris database, data is logically described in the form of a table, that is, a parallel analysis data table, which is used to store and manage data. A table includes rows and columns. Row is a row of data of the user, and Column is used to describe different fields in a row of data.

[0162] In this embodiment, the Spark Doris Connector can be used to read the data in the first transactional relational data table and / or the data in the second transactional relational data table in the MySQL database, and synchronously write them into the first parallel analysis data table and the second parallel analysis data table in the Doris database respectively.

[0163] It is understandable that since parallel analysis databases have parallel processing capabilities and have stronger computing power and query optimization capabilities than relational databases, they can significantly improve the speed and efficiency of large-scale data analysis, facilitate the processing of complex analysis tasks, and provide more in-depth analysis results.

[0164] Report data can provide visual and actionable data views for process tracking and data analysis, supporting decision making and process optimization.

[0165] As an example, the setting statement of the first parallel analysis data table is as follows:

[0166] CREATE RESOURCE "VmRSS_s3"

[0167] PROPERTI

[0168] "type" = "s3",

[0169] "s3.endpoint" = "****",

[0170] "s3.region" = "b**",

[0171] "s3.bucket" = "new-user-bucket",

[0172] "s3.root.path" = "path / to / root / *** / example1",

[0173] "s3.access_key" = "**",

[0174] "s3.secret_key" = "***",

[0175] "s3.connection.maximum" = "50",

[0176] "s3.connection.request.timeout" = "3000",

[0177] "s3.connection.timeout" = "1000" );

[0179] CREATE STORAGE POLICY VmRSS_policy

[0180] PROPERTIES(

[0181] "storage_resource" = "VmRSS_s3",

[0182] "cooldown_ttl" = "30d" / / Cooling time 30 days );

[0184] The setting statement for the second parallel analysis data table is as follows:

[0185] CREATE RESOURCE "size_s3"

[0186] PROPERTI

[0187] "type" = "s3",

[0188] "s3.endpoint" = "****",

[0189] "s3.region" = "b**",

[0190] "s3.bucket" = "new-user-bucket",

[0191] "s3.root.path" = "path / to / root / *** / example2",

[0192] "s3.access_key" = "**",

[0193] "s3.secret_key" = "***",

[0194] "s3.connection.maximum" = "50",

[0195] "s3.connection.request.timeout" = "3000",

[0196] "s3.connection.timeout" = "1000" );

[0198] CREATE STORAGE POLICY size_policy

[0199] PROPERTIES(

[0200] "storage_resource" = "size_s3",

[0201] "cooldown_ttl" = "30d" / / Cooling time 30 days );

[0203] It can be understood that by regularly synchronizing the data in the first transactional relational data table and / or the data in the second transactional relational data table to the parallel analysis database and correspondingly generating the first parallel analysis data table and the second parallel analysis data table, it is possible to ensure that the data in the parallel analysis database is up to date so as to enable timely and accurate analysis. By deleting the data that has been synchronized to the parallel analysis database from the relational database, storage space can be freed up to maintain the efficiency and performance of the relational database.

[0204] In one achievable manner, the parallel analysis database sets the hot and cold data stratification, and the method further includes the following steps:

[0205] First, the data is divided into cold data and hot data according to the access frequency of the data in the first parallel analysis data table and the second parallel analysis data table; and then the cold data is moved to the remote object storage.

[0206] In this embodiment, Doris provides hot and cold data tiering technology. Specifically, the freeze time is set at the partition level of the table, that is, a time period is defined, indicating how long the partition will be frozen. When setting the Freeze time, the remote object storage location of the frozen data is also defined, which is usually an object storage compatible with the S3 (Simple Storage Service) protocol or Hadoop Distributed File System (Hadoop Distributed File System, HDFS). On the backend (BE) node, there is a daemon thread responsible for periodically checking which partitions need to be frozen. Once it is determined that a partition needs to be frozen, the thread will upload the data of the partition to the predefined remote storage location.

[0207] Furthermore, S3 RESOURCE is used to store cold data. S3 RESOURCE is a cloud object storage service that can store and protect any amount of data. When creating S3 RESOURCE, the S3 remote link will be verified to ensure that the RESOURCE is created correctly.

[0208] According to the access frequency of data in the first parallel analysis data table and the second parallel analysis data table, the data is divided into cold data and hot data, and the hot data retention period is specified as 30 days, after which the data is moved to the remote object storage.

[0209] It should be noted that this embodiment does not limit the time period for periodically synchronizing the data in the transactional relational data table to the parallel analysis database and the freezing time in the parallel analysis database. It can be adjusted according to actual task requirements. Usually, the synchronization time period is set to 7 days and the freezing time is set to 30 days.

[0210] It should be further explained that the first parallel analysis data table and the second parallel analysis data table are both detail tables, and therefore the set data models are both detail models.

[0211] As an example, the Doris database table structure of the detail model set for the first parallel analysis data table is as follows:

[0212] CREATE TABLE IF NOT EXISTS example1(

[0213] `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Auto-increment id',

[0214] `name` varchar(200) NOT NULL COMMENT 'Type of application or command',

[0215] `ip` varchar(200) DEFAULT NULL COMMENT 'Interface machine IP address',

[0216] `applicationId` varchar(200) NOT NULL COMMENT 'job id',

[0217] `containerid` varchar(200) DEFAULT NULL COMMENT 'containerid',

[0218] `pid` bigint(20) DEFAULT NULL COMMENT 'Process number',

[0219] `ppid` bigint(20) DEFAULT NULL COMMENT 'Parent process ID',

[0220] `VmRSS` varchar(200) DEFAULT NULL COMMENT 'Physical memory',

[0221] `RssAnon` varchar(200) DEFAULT NULL COMMENT 'Anonymous RSS memory',

[0222] `RssFile` varchar(200) DEFAULT NULL COMMENT 'File RSS memory',

[0223] `RssShmem` varchar(200) DEFAULT NULL COMMENT 'Shared memory',

[0224] `insertTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Data storage time',

[0225] `desc` varchar(255) DEFAULT NULL COMMENT 'Description',

[0226] `other` varchar(255) DEFAULT NULL COMMENT 'Reserved field',

[0227] `time` varchar(100) DEFAULT NULL COMMENT 'Collection time', )

[0229] Duplicate KEY(`applicationId`) / / Detailed model

[0230] DISTRIBUTED BY HASH ((`applicationId`) BUCKETS 3

[0231] PROPERTIES(

[0232] "storage_policy" = "VmRSS_policy"

[0233] ); / / Set hot and cold layers

[0234] The Doris database table structure of the detail model set for the second parallel analysis data table is as follows:

[0235] CREATE TABLE IF NOT EXISTS example2(

[0236] `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Auto-increment id',

[0237] `applicationId` varchar(100) NOT NULL COMMENT 'job id',

[0238] `parallelism` varchar(20) NOT NULL COMMENT 'Maximum degree of parallelism',

[0239] `workmanagerSizes` varchar(20) NOT NULL COMMENT 'workmanager physical memory size',

[0240] `desc` varchar(255) DEFAULT NULL COMMENT 'Description',

[0241] `other` varchar(255) DEFAULT NULL COMMENT 'Reserved field',

[0242] `name` varchar(200) NOT NULL COMMENT 'Job name', )

[0244] Duplicate KEY(`applicationId`) / / Detailed model

[0245] DISTRIBUTED BY HASH ((`applicationId`) BUCKETS 3

[0246] PROPERTIES(

[0247] "storage_policy" = "size_policy"

[0248] ); / / Set hot and cold layers

[0249] It can be understood that through the freezing mechanism, infrequently accessed partitions are marked as cold data for subsequent migration to remote storage. Hot and cold tiering supports all Doris functions. By moving cold data to remote object storage, storage costs can be significantly reduced compared to high-performance local storage, while ensuring minimal loss of query performance. It can still efficiently process analysis tasks without affecting Doris functions.

[0250] In one possible implementation, the specific implementation of determining the information related to the physical memory allocation of the Spark job can be carried out through the following steps:

[0251] First, according to the application program identifier, an application program interface corresponding to the application program identifier is determined; then, according to the application program interface and a preset second file path, information related to allocated physical memory of the Spark job is determined.

[0252] The application programming interface refers to an application programming interface (API) for accessing and managing Spark jobs. In this embodiment, Spark provides a REST API, so that the detailed information of the job, such as the status and configuration information of the application, can be obtained by accessing the REST API of Spark through an HTTP request (such as a GET request) based on the application programming interface.

[0253] The preset second file path may refer to a file path for storing Spark job configurations or logs, which is usually in the file system of the cluster. In this embodiment, the file path is http: / / *** / proxy / applicationId / v1 / .

[0254] For example, by combining the data returned by the API with the configuration information in the file path, you can extract information related to the allocated physical memory of the Spark job, such as Executor memory configuration, Driver memory configuration, the maximum parallelism of the current job, and the physical memory size of a single worker of the Spark job (workmanagerSizes).

[0255] It is understandable that through the application program interface, the current configuration and status information of the Spark job can be accurately obtained to ensure the accuracy and real-time nature of the data. Furthermore, understanding the memory allocation of each job can help identify bottlenecks and optimization opportunities in resource usage, and support more effective resource management and scheduling.

[0256] The Spark memory utilization determination method provided in the embodiment of the present application can ensure the integrity and consistency of data and prevent data loss or damage by storing the information related to the use of physical memory and the information related to the allocation of physical memory in the first transactional relationship data table and the second transactional relationship data table respectively. By synchronizing the data in the transactional relationship data table to the parallel analysis database and setting a hot and cold stratification mechanism, it can not only improve the data analysis efficiency and enhance the data analysis capability, but also optimize the storage cost and improve the response speed and efficiency of the system.

[0257] Based on the above embodiments, Figure 4 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 4 .like Figure 4 As shown, in this embodiment Figure 1 Based on the embodiment, S104 is described in detail, which specifically includes the following steps:

[0258] S401. Determine the total physical memory used by the Spark job according to the used physical memory.

[0259] In this embodiment, the used physical memory VmRSS_t1...N of the processes in N Worker nodes under the same Spark job is collected, and the used physical memory of these processes is added together to obtain the total physical memory VmRSS_app used by the current job, as shown in the formula: VmRSS_t_total=VmRSS_t1+VmRSS_t2+......+VmRSS_tN.

[0260] S402: Determine the total physical memory allocated for the Spark job according to the physical memory of each working node in the Spark job and the number of working nodes in the Spark job.

[0261] In this embodiment, in Spark, each worker node has an allocated physical memory amount, which is specified by the Spark configuration parameters. By multiplying the physical memory with the number of worker nodes, the total physical memory allocated for the Spark job is obtained, as shown below:

[0262] The physical memory of a single worker node is virtual_workmanagerSizes=workmanagerSizes;

[0263] The formula for the number of working nodes is parallelism;

[0264] The total physical memory allocated to a Spark job is the maximum amount of memory configured for a Spark job, as shown in the formula: virtual_size_max=virtual_workmanagerSizes*(parallelism).

[0265] S403: Determine the memory utilization of the Spark job according to the total physical memory used by the Spark job and the total physical memory allocated to the Spark job.

[0266] In this embodiment, the memory utilization rate of the Spark job can be obtained by dividing the total physical memory used by the Spark job VmRSS_app by the total physical memory allocated to the Spark job (the maximum memory configured for the Spark job): virtual_size_max virtual_memory_rate=VmRSS_app / virtual_size_max.

[0267] In some embodiments, the memory utilization of the Spark job can also be stored in the third transactional relational data table, that is, database MySQL table 3, to ensure data integrity. The structure of table 3 is as follows:

[0268] CREATE TABLE `Table 3 (

[0269] `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Auto-increment id',

[0270] `name` varchar(200) NOT NULL COMMENT 'Job name',

[0271] `application_id` varchar(100) NOT NULL COMMENT 'Job id',

[0272] `virtual_size_max` varchar(20) NOT NULL COMMENT 'Maximum physical memory allocated',

[0273] `VmRSS_app` varchar(20) NOT NULL COMMENT 'Total physical memory currently used',

[0274] `virtual_memory_rate` varchar(20) NOT NULL COMMENT 'Physical memory usage',

[0275] `desc` varchar(255) DEFAULT NULL COMMENT 'Description',

[0276] `other` varchar(255) DEFAULT NULL COMMENT 'Reserved field'

[0277] `insertTime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Data entry time'

[0278] PRIMARY KEY (`id`)

[0279] ) ENGINE=InnoDB AUTO_INCREMENT=01 DEFAULT CHARSET=utf8mb4;

[0280] The Spark memory utilization determination method provided in the embodiment of the present application combines the collected fine-grained physical memory to calculate the complete total used physical memory and allocated total physical memory of the Spark job, thereby calculating the accurate memory utilization of the Spark job. By understanding the actual memory usage, the memory configuration parameters of the Spark job can be better adjusted to match the actual needs.

[0281] Based on the above embodiments, Figure 5 Schematic diagram of the process of determining Spark memory utilization provided in the embodiment of the present application Figure 5 .like Figure 5 As shown, the method may include the following steps:

[0282] S501. Obtain a list of Spark jobs to be collected by reading Stream Park, including ApplicationId and job name fields.

[0283] S502: When i=0, determine whether i is less than the list length.

[0284] Among them, i is the index variable used for iteration, which is usually used to traverse the elements in the list and check whether the current index i is within the range of the list to ensure that it does not exceed the list boundary, thereby ensuring that the loop is only performed within the valid job list range.

[0285] S503: If not, end the process.

[0286] It should be noted that if i is not less than the list length, it means that all jobs have been processed and the process ends, thus preventing out-of-bounds errors and ensuring the correct end of the process.

[0287] S504: If yes, collect the physical memory of the current process of the job on Linux.

[0288] S505. Store the data details of the physical memory of the current process of the job into Mysql to generate Table 1.

[0289] S506: Calculate the total physical memory currently used according to all Spark processes of the current job.

[0290] S507. Collect the allocated memory through the REST API provided by Spark.

[0291] S508. Store the data in the allocated memory into Mysql to generate Table 2.

[0292] S509: Calculate and allocate the maximum physical memory.

[0293] S510: Divide the total physical memory currently in use by the maximum allocated physical memory to obtain a physical memory usage rate.

[0294] S511. Every 7 days, store the detailed data of Table 1 and Table 2 in MySQL into Doris database, and delete the data of Table 1 and Table 2 simultaneously.

[0295] S512, storing the calculation result of the physical memory usage data into Mysql, generating Table 3, and adding 1 to the number i, and re-executing S502 to S512.

[0296] In which, after processing the current job, i is increased by 1 so that the next job in the list can be processed in the next loop, thus ensuring that every job is processed.

[0297] The Spark memory utilization determination method provided in the embodiment of the present application can effectively monitor and analyze the memory usage of Spark jobs by setting a loop processing process of traversing the list, thereby optimizing resource allocation and improving job performance and system stability.

[0298] Figure 6 This is a schematic diagram of the structure of the Spark memory utilization determination device provided in the embodiment of the present application. Figure 6 As shown, the Spark memory utilization determination device 60 provided in this embodiment includes:

[0299] An acquisition module 601 is used to acquire an application program identifier corresponding to a Spark job;

[0300] A first determination module 602 is used to determine a node process identifier corresponding to a Spark job in each cluster node according to the application identifier, each cluster node being a running node of the Spark job;

[0301] The reading module 603 is used to read the used physical memory corresponding to the node process stored in the system file according to the node process identifier and based on the system file path;

[0302] The second determining module 604 is used to determine the memory utilization of the Spark job according to the used physical memory and the total physical memory allocated to the Spark job.

[0303] Optionally, the reading module 601 is specifically configured to:

[0304] According to the node process identifier, determine the anonymous resident set size memory, the file resident set size memory and the shared resident set size memory of each process;

[0305] Determines the used physical memory of the node process based on the anonymous resident set size memory, file resident set size memory, and shared resident set size memory of each process.

[0306] Optionally, after determining the used physical memory of the node process, the reading module 603 is further used to:

[0307] Determine the information related to the use of physical memory of the node process, the information related to the use of physical memory includes the node process identifier, the parent node process identifier, the anonymous resident set size process, the file resident set size process, the shared resident set size process, and the used physical memory of the node process; store the information related to the use of physical memory of the node process in the first transactional relationship data table;

[0308] And / or, determining information related to allocated physical memory of the Spark job, where the information related to allocated physical memory includes an application identifier, a maximum degree of parallelism, and physical memory of each working node in the Spark job; and storing the information related to allocated physical memory of the Spark job in a second transactional relational data table.

[0309] Optionally, the reading module 603 is further used for:

[0310] The data in the first transactional relationship data table and / or the data in the second transactional relationship data table are periodically synchronized to the parallel analysis database, and a first parallel analysis data table and a second parallel analysis data table are correspondingly generated to form report data, which is used for process tracking and data analysis;

[0311] Correspondingly, the synchronized data in the first transactional relationship data table and the second transactional relationship data table are deleted.

[0312] Optionally, the parallel analysis database sets the hot and cold data stratification, and the reading module 603 is further used for:

[0313] According to the access frequency of the data in the first parallel analysis data table and the second parallel analysis data table, the data is divided into cold data and hot data;

[0314] Move cold data to remote object storage.

[0315] Optionally, to determine information related to allocated physical memory of a Spark job, the reading module 603 is further used to:

[0316] According to the application program identifier, determining the application program interface corresponding to the application program identifier;

[0317] According to the application program interface and the preset second file path, the information related to the allocated physical memory of the Spark job is determined.

[0318] Optionally, the second determining module 604 is specifically configured to:

[0319] Based on the used physical memory, determine the total physical memory used by the Spark job;

[0320] Determine the total physical memory allocated for the Spark job based on the physical memory of each worker node in the Spark job and the number of worker nodes in the Spark job;

[0321] Determine the memory utilization of the Spark job based on the total physical memory used by the Spark job and the total physical memory allocated to the Spark job.

[0322] The Spark memory utilization determination device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0323] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 7As shown, the electronic device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0324] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0325] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0326] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0327] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0328] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0329] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0330] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0331] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0332] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0333] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0334] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0335] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0336] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0337] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0338] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for determining Spark memory utilization, characterized in that: include: Get the application ID corresponding to the Spark job; According to the application identifier, a node process identifier corresponding to the Spark job is determined in each cluster node, and each cluster node is a running node of the Spark job; According to the node process identifier and based on the system file path, reading the used physical memory corresponding to the node process stored in the system file; A memory utilization rate of the Spark job is determined according to the used physical memory and the total physical memory allocated to the Spark job.

2. The method according to claim 1, characterized in that The step of reading the used physical memory corresponding to the node process stored in the system file according to the node process identifier and based on the system file path includes: Determine the anonymous resident set size memory, the file resident set size memory and the shared resident set size memory of each process according to the node process identifier; The used physical memory of the node process is determined according to the anonymous resident set size memory, the file resident set size memory and the shared resident set size memory of each process.

3. The method according to claim 2, characterized in that After determining the used physical memory of the node process, the method further includes: Determine information related to the use of physical memory of the node process, the information related to the use of physical memory includes the node process identifier, the parent node process identifier, the anonymous resident set size process, the file resident set size process, the shared resident set size process, and the used physical memory of the node process; store the information related to the use of physical memory of the node process in the first transactional relationship data table; And / or, determine the information related to the allocated physical memory of the Spark job, the information related to the allocated physical memory includes the application identifier, the maximum degree of parallelism, and the physical memory of each working node in the Spark job; store the information related to the allocated physical memory of the Spark job in the second transactional relational data table.

4. The method according to claim 3, characterized in that: The method further comprises: The data in the first transactional relationship data table and / or the data in the second transactional relationship data table are periodically synchronized to the parallel analysis database, and a first parallel analysis data table and a second parallel analysis data table are correspondingly generated to form report data, and the report data is used for process tracking and data analysis; Correspondingly, the synchronized data in the first transactional relationship data table and the second transactional relationship data table are deleted.

5. The method according to claim 4, characterized in that The parallel analysis database is configured with hot and cold data stratification, and the method further comprises: According to the access frequency of data in the first parallel analysis data table and the second parallel analysis data table, the data is divided into cold data and hot data; The cold data is moved to a remote object store.

6. The method according to claim 3, characterized in that The determining of information related to the allocated physical memory of the Spark job includes: Determine, according to the application program identifier, an application program interface corresponding to the application program identifier; According to the application program interface and the preset second file path, information related to allocated physical memory of the Spark job is determined.

7. The method according to any one of claims 1 to 6, characterized in that The determining the memory utilization of the Spark job according to the used physical memory and the total physical memory allocated to the Spark job includes: Determine the total physical memory used by the Spark job according to the used physical memory; Determine the total physical memory allocated for the Spark job according to the physical memory of each working node in the Spark job and the number of working nodes in the Spark job; The memory utilization of the Spark job is determined according to the total physical memory used by the Spark job and the total physical memory allocated to the Spark job.

8. A Spark memory utilization determination device, characterized in that: include: The acquisition module is used to obtain the application ID corresponding to the Spark job; A first determination module, configured to determine, according to the application identifier, a node process identifier corresponding to the Spark job in each cluster node, each cluster node being a running node of the Spark job; A reading module, configured to read the used physical memory corresponding to the node process stored in the system file according to the node process identifier and based on the system file path; The second determining module is used to determine the memory utilization of the Spark job according to the used physical memory and the total physical memory allocated to the Spark job.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images