Memory analysis method, device and system

Abstract

The present invention discloses a memory analysis method, device and system. The method comprises: obtaining a memory snapshot file when the system is running, the memory snapshot file comprises at least one type of memory object, each type of memory object comprises attribute information organized according to a predetermined data specification; creating at least one table structure, each table structure corresponds to a type of memory object; parsing the memory snapshot file, and parsing the attribute information of each memory object into a corresponding table structure according to the corresponding data specification; searching at least one table structure, so as to perform memory analysis according to the search result. The method collects the attribute information of the memory objects in the memory snapshot file into multiple table structures, so as to analyze and locate the memory problem by searching the data in the table structure.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a memory analysis method, device and system. Background Art

[0002] Object-oriented language is a universal high-level language and is often used to build various complex systems, such as large e-commerce websites. These systems often need to process various network requests from thousands or even hundreds of millions of users in real time. In order to process these network requests, the running system needs to maintain a large number of objects. In complex systems, various problems often occur in the system, among which memory problems are often common. Memory problems mainly include memory leaks and memory overflows. Memory leaks are caused by the unreasonable organization of code, which leads to the failure to release certain objects in memory in time. Memory overflows are caused by excessive instantaneous access traffic and other reasons, which cause the system process to instantly generate a large number of objects and require a large amount of memory. At this time, the memory is instantly consumed, resulting in server downtime and other failures. If a memory problem occurs, it will have a great impact on the system and ultimately affect the user experience.

[0003] There are some analysis tools on the market for memory issues. The most commonly used ones are jmap and jhat tools that come with JDK. jmap can dump memory snapshot files, which are records of the memory status at a certain moment when the system is running, and jhat can analyze these dumped memory snapshot files. In addition, there are many other commercial tools, such as Eclipse's MemoryAnalyze Tool (MAT). These tools are very helpful to developers. However, most of these tools have some shortcomings. For example, these tools only observe objects at a specific time point when the system is running, and cannot analyze the changes of objects over time. For another example, it is difficult for these tools to perform correlation analysis on multiple objects. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a memory analysis method and system to provide correlation analysis capabilities that existing analysis tools do not have.

[0005] To achieve this objective, according to a first aspect of the present invention, an embodiment of the present invention provides a memory analysis method, comprising:

[0006] Acquire a memory snapshot file when the system is running, the memory snapshot file including at least one type of memory object, each type of memory object including attribute information organized according to a predetermined data specification;

[0007] Create at least one table structure, each table structure corresponds to a type of memory object;

[0008] Parsing the memory snapshot file, and parsing the attribute information of each memory object into a corresponding table structure according to the corresponding data specification;

[0009] The at least one table structure is retrieved to facilitate memory analysis based on the retrieval result.

[0010] Optionally, the at least one type includes a class object and an instance object, and parsing the attribute information of each memory object into a corresponding table structure includes:

[0011] Parsing the attribute information of the class object in the memory snapshot file into a class table;

[0012] Parse the attribute information of the instance object in the memory snapshot file into the instance table.

[0013] Optionally, the searching of the at least one table structure includes: selecting at least two items from the class table, the instance table and other data structures for associated searching.

[0014] Optionally, the following search results for memory analysis are obtained based on the association search between the class table and the instance table:

[0015] The names of several class objects with the most instance objects;

[0016] The names of the class objects corresponding to the instance objects that occupy the most memory;

[0017] Statistics of the instance object.

[0018] Optionally, the retrieving the at least one table structure includes:

[0019] receiving an SQL statement, the SQL statement being used to retrieve data from the at least one table structure;

[0020] According to the semantics of the SQL statement, n levels of tasks are constructed, each level of tasks contains one or more tasks, tasks at the same level are executed in parallel, and tasks at different levels are executed serially, the first level of tasks obtains data from the at least one table structure, the second to n-1 level tasks receive the output data of the previous level of tasks and output it to the next level of tasks, and the nth level of tasks summarizes the data as a response to the SQL statement, where n is greater than 1.

[0021] Optionally, a correspondence between the at least one table structure and the at least one type of memory object is established through configuration information.

[0022] Optionally, memory snapshot files of multiple different moments when the system is running are obtained, and for each memory snapshot file at a different moment, attribute information of each memory object contained therein is parsed into a corresponding table structure, and at least one of the table structures is retrieved to obtain status information of each memory object at multiple different moments, so as to analyze changes of each memory object at multiple different moments.

[0023] Optionally, the system is executed on a virtual machine, and the memory snapshot file is a memory state file of the virtual machine at a certain moment.

[0024] Optionally, the virtual machine is a JAVA virtual machine, and a memory snapshot file of the virtual machine is obtained through a DUMP command of the JAVA virtual machine, or a memory snapshot file of a heap of the JAVA virtual machine is obtained through a DUMP command of the JAVA virtual machine.

[0025] Optionally, the method is implemented using a data lake system.

[0026] Optionally, the at least one type includes one or more of the following types: class, instance, data structure, array, pointer.

[0027] In a second aspect, an embodiment of the present invention provides a memory analysis device, including:

[0028] A data preparation module, which obtains a memory snapshot file when the system is running, wherein the memory snapshot file includes at least one type of memory object, and each type of memory object includes attribute information organized according to a predetermined data specification;

[0029] A table creation module is used to create at least one table structure, each table structure corresponds to a type of memory object;

[0030] A data parsing module, used to parse the memory snapshot file and parse the attribute information of each memory object into a corresponding table structure according to the corresponding data specification;

[0031] The retrieval analysis module is used to retrieve the at least one table structure so as to perform memory analysis based on the retrieval results.

[0032] Optionally, the at least one type includes a class object and an instance object, and the data parsing module includes:

[0033] Parsing the attribute information of the class object in the memory snapshot file into a class table;

[0034] Parse the attribute information of the instance object in the memory snapshot file into the instance table.

[0035] Optionally, the retrieval analysis module includes: selecting at least two items from the class table, the instance table and other data structures for associated retrieval.

[0036] Optionally, the retrieval analysis module obtains the following retrieval results for memory analysis:

[0037] The names of several class objects with the most instance objects;

[0038] The names of the class objects corresponding to the instance objects that occupy the most memory;

[0039] Statistics for all instance objects.

[0040] Optionally, the retrieval analysis module includes: receiving an SQL statement, the SQL statement being used to retrieve data from the at least one table structure; constructing n levels of tasks according to the semantics of the SQL statement, each level of tasks containing one or more tasks, tasks at the same level being executed in parallel, and tasks at different levels being executed serially, the first level of tasks acquiring data from the at least one table structure, the second to n-1 level tasks receiving output data from the previous level of tasks and outputting it to the next level of tasks, the nth level of tasks summarizing the data as a response to the SQL statement, where n is greater than 1.

[0041] Optionally, the memory analysis device further includes: a configuration module, configured to establish configuration information, wherein the configuration information indicates a correspondence between the multiple table structures and the multiple types of memory objects.

[0042] Optionally, the data preparation module obtains memory snapshot files at multiple different times when the system is running, and the data parsing module parses the attribute information of each memory object contained in the memory snapshot file at each different time into a corresponding table structure. The retrieval and analysis module retrieves at least one table structure to obtain the status information of each memory object at multiple different times, so as to analyze the changes of each memory object at multiple different times.

[0043] Optionally, the system is executed on a virtual machine, the virtual machine is a JAVA virtual machine, and the data preparation module obtains a memory snapshot file of the virtual machine through a DUMP command of the JAVA virtual machine, or obtains a memory snapshot file of the heap of the JAVA virtual machine through a DUMP command of the JAVA virtual machine.

[0044] Optionally, the at least one type includes one or more of the following types: class, instance, data structure, array, pointer.

[0045] In a third aspect, the present invention provides a memory analysis system, comprising a plurality of front nodes, a plurality of computing nodes and a plurality of data sources, wherein the plurality of front nodes receive various data requests from applications and distribute the various data requests to the plurality of computing nodes, and the plurality of computing nodes process various data requests, wherein the data requests include SQL query requests related to memory analysis,

[0046] At least one computing node of the plurality of computing nodes retrieves at least one table structure in the relational data source according to the SQL query request, so as to perform memory analysis according to the retrieval result,

[0047] The at least one computing node is further configured to execute:

[0048] Acquire a memory snapshot file when the system is running and store it in an object data source, wherein the memory snapshot file includes at least one type of memory object, and each type of memory object includes attribute information organized according to a predetermined data specification;

[0049] Creating at least one table structure of the relational data source, each table structure corresponds to a type of memory object;

[0050] The memory snapshot file is parsed, and the attribute information of each memory object is parsed into a corresponding table structure according to a corresponding data specification.

[0051] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory further stores computer instructions executable by the processor, and when the computer instructions are executed, any of the methods described above is implemented.

[0052] In a fifth aspect, an embodiment of the present invention provides a computer-readable medium, wherein the computer-readable medium stores computer instructions executable by an electronic device, and when the computer instructions are executed, any of the methods described above is implemented.

[0053] The memory analysis method, device and system provided by the embodiment of the present invention parse the attribute information of the memory object from the memory snapshot file and store it in the corresponding table structure, so as to analyze and locate the memory problem by retrieving the data in the table structure. Furthermore, the data in multiple table structures can be associated and retrieved, and the memory problem can be determined according to the result of the associated retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and other objects, features and advantages of the present invention will become more apparent through the description of the embodiments of the present invention with reference to the following drawings, in which:

[0055] Figure 1 is a schematic diagram of an exemplary JAVA virtual machine memory;

[0056] Figure 2 is a flowchart of a memory analysis method provided according to an embodiment of the present invention;

[0057] Figure 3 shows the data records of the class table and the instance table;

[0058] Figure 4 is a structural diagram of a memory analysis device provided according to an embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of the multi-level tasks for processing SQL statements;

[0060] Figure 6a is a schematic diagram of the software structure of an exemplary data lake system;

[0061] Figure 6b Show Figure 6a A deployment diagram of a data lake service 602;

[0062] Figure 7 is a schematic diagram of an electronic device for implementing the method and system of an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0064] Below, before describing the embodiments of the present invention in detail based on the accompanying drawings, some aspects of the virtual machine interpretation and execution system are first introduced by taking a JAVA virtual machine as an example.

[0065] When executing a JAVA program, the JAVA virtual machine divides the memory it manages into several different data areas. These data areas have their own uses and creation and destruction times. Some data areas exist when the virtual machine process is started, while some areas are created and destroyed depending on the start and end of the user thread. Figure 1 This is a schematic diagram of an exemplary JAVA virtual machine memory. As shown in the figure, these data areas include method area, virtual machine stack, local method stack, heap, and program counter.

[0066] The program counter is a small piece of memory space, which can be regarded as a line number indicator of the bytecode executed by the current thread (JAVA source code must be converted into bytecode to be executed in the virtual machine). Since the multi-threading of the JAVA virtual machine is implemented by switching threads in turn and allocating processor execution time, at any given moment, a processor (a core for a multi-core processor) will only execute instructions in one thread. Therefore, in order to restore to the correct execution position after the thread switch, each thread needs to have an independent program counter. The counters between threads do not affect each other and are stored independently, so the program counter is a "thread-private" memory space.

[0067] The virtual machine stack describes the memory space for JAVA method execution: when each method is executed, a stack frame is created to store local variable table, operation stack, dynamic link, method exit and other information. The process from each method being called to the completion of execution corresponds to the process of a stack frame being pushed into and popped from the virtual machine stack. The virtual machine stack is thread-private. At present, some virtual machines directly combine the local method stack and the virtual machine stack into one.

[0068] The heap is a memory area shared by all threads. The purpose of the heap is to store object instances, and almost all object instances are allocated memory in the heap. However, with the development of compilation technology, the principle of allocating object instances on the heap is also changing.

[0069] The method area is a memory area shared by all threads. It is used to store class information, constants, static variables, and code compiled by the just-in-time compiler that have been loaded by the virtual machine. The runtime constant pool is part of the method area and is used to store various literals and symbol references generated during compilation.

[0070] Although the memory of various virtual machines is divided into multiple data areas, different types and versions of virtual machines will affect the way the data areas are divided. For example, the JAVA virtual machine divides the local method stack and the virtual machine stack differently due to the different JDK versions used. For another example, the data areas divided by the Sun HotSpot virtual machine are also different from those of the JAVA virtual machine. In this regard, a practical method is to read the corresponding virtual machine specifications to learn how the data areas are divided and the data specifications of each data area.

[0071] Hereinafter, embodiments of the present invention will be described in detail based on the accompanying drawings.

[0072] Figure 21 is a flow chart of a memory analysis method provided according to an embodiment of the present invention, which specifically includes the following steps.

[0073] In step S211, a memory snapshot file during system runtime is obtained.

[0074] The system can be any system, such as a social system, a monitoring system, a banking system, a telecommunications system, an office system, and so on. A memory snapshot file refers to a record of the memory state at a certain moment when the system is running. For a system programmed in an object-oriented language, a memory snapshot file contains one or more types of memory objects. A memory object refers to a storage area in the computer memory of a variable body constructed according to various data types supported by an object-oriented programming language. The type of a memory object is the data type used to construct a memory object. Accordingly, in this article, memory objects constructed according to different data types such as classes, instances, data structures, arrays, and pointers can be referred to as class objects, instance objects, data structure objects, array objects, and pointer objects. Each type of memory object includes attribute information organized according to a predetermined data specification.

[0075] Systems programmed in object-oriented languages ​​are generally interpreted and executed by virtual machines, so collecting the memory snapshot files of virtual machines can obtain the status information of all memory objects of the system at a certain moment. However, due to the memory recovery mechanism of the virtual machine, most memory objects are created and recovered by the virtual machine, and only a small number of memory objects cannot be recovered by the virtual machine due to problems such as wrong dependencies. Therefore, the acquisition of most memory objects is not of great value to the memory analysis of the virtual machine, and the huge amount of data will increase the workload. Therefore, for this step, it is preferred to only collect memory objects with a higher probability of error for analysis. Due to the complex association relationship between classes, the probability that the instance object cannot be recovered by the virtual machine is high, so it is preferred to only collect instance objects for analysis. Further, since the instance object is generally created on the heap of the virtual machine, the memory snapshot file of this step can only be the memory snapshot file of the heap. Of course, if class information and instances are still needed for comparison, memory snapshot files in other areas can continue to be obtained.

[0076] Take the JAVA virtual machine as an example. The JAVA virtual machine interprets and executes the JAVA system. You can use the JAVA dump command to obtain the memory snapshot file of the JAVA virtual machine. The example of the JAVA dump command is: sudo-u admin jmap-dump:format=b,file= / home / admin / zhujl.dump 15065, 15065 is the process number, / home / admin / zhujl.dump is the output file name, and b indicates that the file format is binary. For binary dump files, you need to convert them to decimal before identifying them.

[0077] Of course, if the system does not use a virtual machine to interpret and execute, other means may be used to obtain the memory snapshot file, such as using an operating system interface to obtain the memory snapshot file.

[0078] In step S212, at least one table structure is created. In this step, a library structure may be constructed first, and then one or more table structures may be constructed under the library structure. The library structure is used to specify the storage location of the table structure. Each table structure corresponds to a type of memory object. As can be seen from the above, a data type as a type of memory object, i.e., a class, an instance, a data structure, an array, a pointer, etc. may all be used as a type of memory object. Therefore, after determining which types of memory objects need to be based on for memory analysis, multiple table structures are created for these types, and each table structure stores the attribute information of the memory object of that type.

[0079] It should be pointed out that the table structure in the present invention can be a database table created using an existing database management system, or it can be a table structure that is not constructed using an existing database management system. For example, a self-developed data analysis system is used to create a table structure in memory or a file system. The organizational form of the data analysis system can be similar to an existing database management system, and standard SQL statements can be used to reduce the difficulty of use for technical personnel.

[0080] In step S213, the memory snapshot file is parsed, and the attribute information of each memory object is parsed into a corresponding table structure according to the corresponding data specification. If the memory snapshot file is a memory snapshot file of a virtual machine, the virtual machine specification can be used to first determine the storage location of each memory object in the memory snapshot file, and then, starting from the corresponding storage location of the memory snapshot file, the attribute information of each memory object is parsed according to the data specification of each memory object and added to the corresponding table structure.

[0081] In step S214, at least one table structure is searched to facilitate memory analysis based on the search results. The above steps have formed the attribute information of various types of memory objects into data of at least one table structure. In this step, the memory problem is analyzed and located by searching the data in each table structure. The search in this step can be either a single table search or a multi-table association search, with the goal of determining the memory problem as quickly as possible.

[0082] In this embodiment, the attribute information of the memory object is parsed from the memory snapshot file and stored in the corresponding table structure, so that the memory problem can be analyzed and located by retrieving the data in the table structure. Furthermore, the data in multiple table structures can be associated and retrieved, and the memory problem can be determined based on the results of the associated retrieval.

[0083] As an optional embodiment, only two types of memory objects are collected from the memory snapshot file: class objects and instance objects. Specifically, a class table is created for the class object, and an instance table is created for the instance object. Then, the attribute information of the class object is parsed from the memory snapshot file according to the data specifications of the class object and stored in the class table. The attribute information of the instance object is parsed from the memory snapshot file according to the data specifications of the instance object and stored in the instance table. Then, various instance tables and class tables are retrieved, and the memory problem is analyzed and located according to the retrieval results. In object-oriented programming languages, the relationship between classes includes inheritance, implementation, association, dependency, aggregation, etc. Instances are data types constructed based on classes. The memory problem of an instance will affect other instances or other structures. For example, an instance of a class A applies for a section of memory on the heap of a virtual machine, but it is not released. Then this memory problem will affect instances created by other classes that have an association relationship with the class A. Therefore, it is very valuable to analyze memory problems based on instances and classes.

[0084] As an optional embodiment, memory snapshot files of multiple different times when the system is running can be obtained. For each memory snapshot file at a different time, the attribute information of each memory object contained therein can be parsed into a corresponding table structure. The status information of each memory object at multiple different times can be obtained by retrieving the table structure, so as to facilitate comparative analysis of the changes of each memory object at multiple different times.

[0085] As an optional embodiment, the corresponding relationship between at least one table structure and at least one type of memory object is indicated by configuration information, and then step S213 can implement parsing the attribute information of the memory object in the memory snapshot file into the corresponding table structure based on the corresponding relationship. The configuration information can be stored in a file or in a database table.

[0086] As an optional embodiment, a query interface for inputting SQL statements is provided to the user. After receiving the SQL statement provided by the user, the table structure is retrieved according to the SQL statement. If the table structure involved in the SQL statement lacks corresponding data, the memory snapshot file can be read in real time on the designated server and parsed into the table structure to complete the retrieval.

[0087] The following table structure and attached Figure 3 The class table, instance table, data parsing and data retrieval process are introduced in detail.

[0088] Table 1 shows the class table structure table_of_class storing the attribute information of the class object.

[0089] Table 1

[0090]

[0091] Table 2 is a table structure table_of_instant that stores attribute information of instance objects.

[0092] Table 2

[0093]

[0094] The table structures table_of_class and table_of_instant are associated through class identifiers. It should be noted that the above table structures do not include method information because method information is not very useful for virtual machine memory analysis. In addition, if the method contains memory objects such as classes, instances, data structures, arrays, pointers, etc., these memory objects will also be collected and stored in the corresponding table structures.

[0095] Based on the class table and instance table, we can get Figure 3 As shown in the figure, the id and class_id in the instance table and class table are assigned during the system operation and serve as the unique identifier of the corresponding record. By associating the classID field of the instance table with the id field of the class table, the statistical information of the instances related to the class can be obtained, and then the user can analyze the statistical information to analyze and locate memory problems.

[0096] Based on the class table and the instance table, an associated search can be performed. For example, the table structures table_of_class and table_of_instant can be associated through the class identifier, and the associated data can be grouped, deduplicated, and arranged in descending order to obtain the class names of the ten classes that generate the most instance objects. Generally speaking, the more instances a class creates, the more complex the dependencies, and the more likely it is to cause memory problems. Therefore, analyzing these classes helps to locate memory problems as soon as possible. For another example, the table structures table_of_class and table_of_instant can be jointly queried through the class identifier, and the obtained data can be grouped, sorted, and the like to obtain the names of the 10 classes whose data members occupy the most memory space and the size of the memory space obtained by statistics (expressed in bits). Generally speaking, the memory space occupied by the data members of each class can be obtained by estimation, and the statistical value of the above SQL statement is compared with the estimated value to troubleshoot memory problems. The third example is to jointly query the table structures table_of_class and table_of_instant through the class identifier, and obtain the number of instances created by each class by statistics, and export this statistical data to the table structure target_table. As mentioned above, the more instances a class creates and the more complex its dependencies are, the more likely it is to have memory problems. Therefore, focusing on analyzing classes that create more instances will help locate memory problems as quickly as possible.

[0097] Of course, the association query that the present invention can provide is not limited to the above examples. Moreover, not only can the table structure of the same memory snapshot file be associated with the query, but also the table structure of multiple memory snapshot files can be associated with the query. For example, two systems A and B, A runs on virtual machine v1, and B runs on virtual machine v2. Then, their respective memory snapshot files can be exported respectively, and the associated query can be performed based on the exported table structure. If both systems A and B use instances constructed by the same class, the two instances can be associated by the class name, and then it can be determined whether there are abnormalities in the two instances. This method can help troubleshoot some problems. In addition, the memory snapshot files of systems A and B can be imported into heterogeneous data sources. For example, the memory snapshot file of system A is imported into the file storage system, and the memory snapshot file of system B is imported into the ORACLE database, and the associated query between the two is realized by programming.

[0098] Figure 4 is a structural diagram of a memory analysis device provided according to an embodiment of the present invention.

[0099] like Figure 4 As shown, the module 40 is divided into a data preparation module 401 , a table creation module 402 , a data parsing module 403 and a retrieval analysis module 404 .

[0100] The data preparation module 401 is used to obtain a memory snapshot file when the system is running, and the memory snapshot file includes at least one type of memory object, and each type of memory object includes attribute information organized according to a predetermined data specification. Specifically, the system 40 may include configuration information, and the configuration information indicates the server IP, port number, and instructions for obtaining the memory snapshot (including input parameters and output parameters) for obtaining the memory snapshot file. Through the above configuration information, the system 40 can actively (regularly or conditionally triggered) obtain the memory snapshot file on the specified server. For example, the system 40 provides the user with an interface for querying each table structure data. When the user submits a user request using the interface, if the data preparation module 401 checks that the corresponding table structure lacks corresponding data, the memory snapshot file is obtained in real time according to the configuration information, and after parsing into the table structure, the query request is executed. The system 40 can also provide the user with an interface for uploading the memory snapshot file, and the user decides when to upload the memory snapshot file to the system.

[0101] The table creation module 402 is used to create at least one table structure, each table structure corresponds to a type of memory object. A data type can be used as the type of the memory object, and the data type is, for example, a class, an instance, a data structure, an array, a pointer, etc., and the memory object is a class object, an instance object, a data structure object, an array object, a pointer object, etc.

[0102] The data parsing module 403 is used to parse the memory snapshot file, and parse the attribute information of each memory object into the corresponding table structure according to the corresponding data specification. The data specification can be created by the developer or provided by the user. For example, when the user uploads the memory snapshot file, he / she submits the data specification corresponding to the memory snapshot file. The data specification should be such information, for example, indicating that the first binary character to the 100th binary character of the memory snapshot file is the attribute information of the class object, and the first binary character to the tenth binary character is the class name, and so on. Through the data specification, the attribute information of the memory object can be parsed, and then the attribute information can be added to the table structure according to the correspondence between the memory object and the table structure.

[0103] The retrieval analysis module 404 is used to retrieve the data in the table structure so as to perform memory analysis based on the retrieval results. Optionally, the retrieval can be implemented through a received user request. The user request generally describes the retrieval method for various table structures, and the user request may also include SQL language. The retrieval analysis module 404 executes the SQL statement to obtain the retrieval results, and then the user analyzes the retrieval results to locate the memory problem.

[0104] In this embodiment, the attribute information of the memory object is parsed from the memory snapshot file and stored in the corresponding table structure, so that the data in the table structure can be retrieved to determine the memory problem. Furthermore, the data in multiple table structures can be associated and retrieved, and the memory problem can be determined based on the results of the associated retrieval.

[0105] As an optional embodiment, the data parsing module 403 can only parse class objects and instance objects, that is, parse the attribute information of the class objects in the memory snapshot file into the class table, and parse the attribute information of the instance objects in the memory snapshot file into the instance table. The retrieval analysis module 404 only performs association retrieval on the class table and the instance table.

[0106] As an optional embodiment, the class table, instance table and other data sources are also associated and retrieved. For example, the class table, instance table and other data sources are used to obtain the names of several class objects that generate the most instance objects, and the names of class objects corresponding to several instance objects that occupy the most memory.

[0107] As an optional embodiment, the retrieval analysis module 404 constructs n-level tasks according to the semantics of the SQL statements submitted by the user. Each level of tasks contains one or more tasks. Tasks at the same level are executed in parallel, and tasks at different levels are executed in series. The first-level task obtains data from the data source, and the second to n-1-level tasks receive the output data of the previous level task and output it to the next level task. The n-level task aggregates the data to respond to the user request, where n is greater than 1. The multi-task processing method helps to improve the retrieval efficiency. Figure 5 Provide a more specific description.

[0108] For any SQL statement, the retrieval analysis module 404 constructs Figure 5 The multi-level tasks shown. As shown in the figure, each node is a task. First, the first-level task obtains data from three data sources respectively, and then the data is filtered, aggregated, combined, sorted, reduced, and other operations are performed through the tasks of the following levels. Finally, the search results that conform to the SQL statement are output. Among them, TableScanOperator represents the task of obtaining data from the data source. This task supports heterogeneous data sources. For example, data can be obtained from relational databases or memory snapshot files; FilterOperator represents the task of filtering data, that is, filtering the data output by TableScanOperator once and retaining only part of the data; ProjectOperator is the task of mapping data; JoinOperator is the task of combining data; AggregationOperator is the task of aggregating data; sortOperator is the task of sorting data; LimitOperator is the task of limiting data.

[0109] As an optional embodiment, the system further includes a configuration module for establishing configuration information, wherein the configuration information indicates the correspondence between multiple table structures and multiple types of memory objects. The configuration information can be stored in a file or in a database table.

[0110] As an optional embodiment, the above system is formed into a WEB system, and provides a query interface for users to input SQL statements. After receiving the SQL statement provided by the user, the system triggers the execution of the data preparation module and the data parsing module to prepare the data, and then retrieves the data according to the SQL statement, and returns the result to the system, so that data can be obtained in real time according to demand. Of course, if the amount of data is relatively large, the preferred method is to prepare the data and parse it into a table structure before providing it to the user for retrieval.

[0111] As an optional implementation, the above method and system can be implemented on a data lake (DLA) system. The data lake system is a large warehouse that stores a variety of raw data of an enterprise, where the data can be accessed, processed, analyzed and transmitted. The data lake system obtains raw data from multiple data sources of the enterprise, and for different purposes, the same raw data may have multiple copies of data that meet specific internal model formats. Therefore, the data processed in the data lake system may be any type of information, from structured data to completely unstructured data. In addition, the data lake system provides an interactive federated query service based on Serverless, which can analyze and integrate data from data sources such as object storage (OSS), databases (PostgreSQL / MySQL, etc.) and NoSQL (TableStore) using standard SQL.

[0112] Figure 6a 60 is a schematic diagram of the structure of an exemplary data lake system. As shown in the figure, the data lake service 602 can collect data from multiple heterogeneous data sources 601 for joint query. The data lake service 602 also provides users with multiple data interfaces 603. Users can submit SQL statements to the data lake service 603 through the multiple data interfaces 603 and obtain search results. At the same time, the data lake server obtains various data from the data interface 603 and stores the various data in the data source 601 after processing. Figure 6b A deployment diagram of a data lake service 602 is shown. As shown in the figure, the data lake service 602 is deployed on a computer cluster 610. The computer cluster 610 includes multiple front nodes FN and multiple computing nodes CN. The front node FN is used to receive various data requests from the application, and distribute the various data requests to multiple computing nodes CN using the configuration metadata Meta to achieve load balancing. The computing node CN actually processes various data requests. The computing node CN can optimize various data requests using the data source metadata Meta. Multiple data sources 601, such as relational data sources (RDS), table storage (Table Store), object storage (OSS), and the like.

[0113] The memory analysis method and device provided by the present invention can be applied to Figure 6bAt least one computing node CN shown. First, the data request forwarded by the front node FN includes an SQL query request related to memory analysis. After the computing node CN receives such a request, it retrieves at least one table structure of the relational data source according to the SQL query request, so that the user can perform memory analysis according to the retrieval result. The computing node CN is also used to parse the memory snapshot file to generate at least one table structure of the relational data source used for retrieval in the above steps. Specifically, the computing node can obtain the memory snapshot file during system runtime via or without the front node FN and store it in the object data source, and then create at least one table structure of the relational data source. Each table structure corresponds to a type of memory object, and then parse the memory snapshot file, and parse the attribute information of each memory object into the corresponding table structure according to the corresponding data specification. It can be seen that the architecture based on the data lake service is conducive to the rapid construction of the memory analysis device provided by the present invention.

[0114] Corresponding to the above embodiment, the present invention also provides an electronic device, such as Figure 7 As shown, at the hardware level, the electronic device 70 includes a memory 703 and a processor 702. In addition, in some cases, it also includes input and output devices 704 and other hardware 701. The memory 703 is, for example, a high-speed random access memory (Random-Access Memory, RAM), or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage device, etc. The input and output devices 704 are, for example, devices such as a display, a keyboard, a mouse, and a network controller. The processor 702 can be built based on various models of processors currently on the market. The processor 702, the memory 703, the input and output devices 704 and other hardware 701 are interconnected through a bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The fact that only one line is used does not mean that there is only one bus or only one type of bus.

[0115] The memory 703 is used to store programs. Specifically, the program may include program code, and the program code includes computer instructions. The memory may include internal memory and non-volatile memory, and provide computer instructions and data to the processor 702. The processor 702 reads the corresponding computer program from the memory 703 into the internal memory and then runs it, forming a database expansion method at the logical level, and is specifically used for the following steps: obtaining a memory snapshot file when the system is running, the memory snapshot file includes at least one type of memory object, and each type of memory object includes attribute information organized according to a predetermined data specification; creating at least one table structure, each table structure corresponds to a type of memory object; parsing the memory snapshot file, and parsing the attribute information of each memory object into a corresponding table structure according to the corresponding data specification; searching the at least one table structure to facilitate memory analysis according to the search results.

[0116] Those skilled in the art will appreciate that the present invention can be implemented as a system, method, and computer program product. Therefore, the present invention can be specifically implemented in the following forms, i.e., complete hardware, complete software (including firmware, resident software, microcode), and can also be implemented in the form of a combination of software and hardware. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains a computer-readable program code.

[0117] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by a processing unit, a device or a device, or used in combination with the program.

[0118] Computer readable signal media may include data signals propagated in baseband or as part of a chopped wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction system, apparatus, or device.

[0119] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0120] The computer program code for executing the embodiment of the present invention can be written in one or more programming languages ​​or combinations. The programming language includes object-oriented programming languages, such as JAVA, C++, and can also include conventional procedural programming languages, such as C. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A memory analysis method, comprising: Acquire a memory snapshot file when the system is running, the memory snapshot file including at least one type of memory object, each type of memory object including attribute information organized according to a predetermined data specification; Create at least one table structure, each table structure corresponds to a type of memory object; each table structure stores attribute information of the type of memory object; Parsing the memory snapshot file, and parsing the attribute information of each memory object into a corresponding table structure according to the corresponding data specification; Data is retrieved from the at least one table structure to facilitate memory analysis based on the retrieval results.

2. The memory analysis method according to claim 1, wherein: The at least one type includes a class object and an instance object, and the parsing of the attribute information of each memory object into a corresponding table structure includes: Parsing the attribute information of the class object in the memory snapshot file into a class table; Parse the attribute information of the instance object in the memory snapshot file into the instance table.

3. The memory analysis method according to claim 2, wherein: The retrieving the at least one table structure includes: selecting at least two items from the class table, the instance table and other data structures for associated retrieval.

4. The memory analysis method according to claim 3, wherein: The following search results for memory analysis are obtained based on the association search between the class table and the instance table: The names of several class objects with the most instance objects; The names of the class objects corresponding to the instance objects that occupy the most memory; Statistics of the instance object.

5. The memory analysis method according to claim 1, wherein searching the at least one table structure comprises: receiving an SQL statement, the SQL statement being used to retrieve data from the at least one table structure; According to the semantics of the SQL statement, n levels of tasks are constructed, each level of tasks contains one or more tasks, tasks at the same level are executed in parallel, and tasks at different levels are executed serially, the first level of tasks obtains data from the at least one table structure, the second to n-1 level tasks receive the output data of the previous level of tasks and output it to the next level of tasks, and the nth level of tasks summarizes the data as a response to the SQL statement, where n is greater than 1.

6. The memory analysis method according to claim 1, wherein: A correspondence between the at least one table structure and the at least one type of memory object is established through configuration information.

7. The memory analysis method according to claim 1, wherein: Obtain memory snapshot files at multiple different times when the system is running. For each memory snapshot file at a different time, parse the attribute information of each memory object contained therein into a corresponding table structure, retrieve at least one of the table structures, and obtain the status information of each memory object at multiple different times, so as to analyze the changes of each memory object at multiple different times.

8. The method according to claim 1, wherein: The system is executed on a virtual machine, and the memory snapshot file is a memory state file of the virtual machine at a certain moment.

9. The method according to claim 8, wherein: The virtual machine is a JAVA virtual machine, and a memory snapshot file of the virtual machine is obtained through a DUMP command of the JAVA virtual machine, or a memory snapshot file of a heap of the JAVA virtual machine is obtained through a DUMP command of the JAVA virtual machine.

10. The method according to claim 1, wherein: The memory analysis method is implemented using a data lake system.

11. The method according to claim 1, wherein: The at least one type includes one or more of the following types: class, instance, data structure, array, pointer.

12. A memory analysis device, comprising: A data preparation module, which obtains a memory snapshot file when the system is running, wherein the memory snapshot file includes at least one type of memory object, and each type of memory object includes attribute information organized according to a predetermined data specification; A table creation module is used to create at least one table structure, each table structure corresponds to a type of memory object; each table structure stores attribute information of the memory object of this type; A data parsing module, used to parse the memory snapshot file and parse the attribute information of each memory object into a corresponding table structure according to the corresponding data specification; The retrieval analysis module is used to retrieve data from the at least one table structure so as to perform memory analysis based on the retrieval results.

13. The memory analysis device according to claim 12, wherein: The at least one type includes a class object and an instance object, and the data parsing module includes: Parsing the attribute information of the class object in the memory snapshot file into a class table; Parse the attribute information of the instance object in the memory snapshot file into the instance table.

14. The memory analysis device according to claim 13, wherein: The retrieval analysis module includes: selecting at least two items from the class table, the instance table and other data structures for associated retrieval.

15. The memory analysis device according to claim 14, wherein: The retrieval analysis module obtains the following retrieval results for memory analysis: The names of several class objects with the most instance objects; The names of the class objects corresponding to the instance objects that occupy the most memory; Statistics for all instance objects.

16. The memory analysis device according to claim 12, wherein the retrieval analysis module comprises: receiving an SQL statement, the SQL statement being used to retrieve data from the at least one table structure; According to the semantics of the SQL statement, n levels of tasks are constructed, each level of tasks contains one or more tasks, tasks at the same level are executed in parallel, and tasks at different levels are executed serially, the first level of tasks obtains data from the at least one table structure, the second to n-1 level tasks receive the output data of the previous level of tasks and output it to the next level of tasks, and the nth level of tasks summarizes the data as a response to the SQL statement, where n is greater than 1.

17. The memory analysis device according to claim 12, further comprising: The configuration module is used to establish configuration information, wherein the configuration information indicates the corresponding relationship between the multiple table structures and the multiple types of memory objects.

18. The memory analysis device according to claim 12, wherein: The data preparation module obtains memory snapshot files at multiple different times when the system is running. The data parsing module parses the attribute information of each memory object contained in the memory snapshot file at each different time into a corresponding table structure. The retrieval and analysis module retrieves at least one table structure to obtain the status information of each memory object at multiple different times, so as to analyze the changes of each memory object at multiple different times.

19. The memory analysis device according to claim 12, wherein: The system is executed on a virtual machine, which is a JAVA virtual machine. The data preparation module obtains a memory snapshot file of the virtual machine through a DUMP command of the JAVA virtual machine, or obtains a memory snapshot file of a heap of the JAVA virtual machine through a DUMP command of the JAVA virtual machine.

20. The system of claim 12, wherein: The at least one type includes one or more of the following types: class, instance, data structure, array, pointer.

21. A memory analysis system, comprising a plurality of front nodes, a plurality of computing nodes and a plurality of heterogeneous data sources, wherein the plurality of front nodes receive various data requests from applications and distribute the various data requests to the plurality of computing nodes, and the plurality of computing nodes process the various data requests, wherein the various data requests include SQL query requests related to memory analysis, At least one computing node of the plurality of computing nodes retrieves at least one table structure in the relational data source according to the SQL query request, so as to perform memory analysis according to the retrieval result, At least one computing node of the plurality of computing nodes is further configured to execute: Acquire a memory snapshot file when the system is running and store it in an object data source, wherein the memory snapshot file includes at least one type of memory object, and each type of memory object includes attribute information organized according to a predetermined data specification; Creating at least one table structure of the relational data source, each table structure corresponds to a type of memory object; each table structure stores attribute information of the type of memory object; The memory snapshot file is parsed, and the attribute information of each memory object is parsed into a corresponding table structure according to a corresponding data specification.

22. An electronic device comprising a memory and a processor, wherein the memory further stores computer instructions executable by the processor, and when the computer instructions are executed, the memory analysis method according to any one of claims 1 to 11 is implemented.

23. A computer-readable medium storing computer instructions executable by an electronic device, wherein when the computer instructions are executed, the memory analysis method according to any one of claims 1 to 11 is implemented.

Images