Core compressed file generation method and device

Abstract

The present invention discloses a method for generating a core compressed file: obtaining the PID of a process that triggers a coredump process, querying maps and smaps information through the process PID to obtain the memory usage and distribution of the process; performing core data interception based on the memory usage and distribution of the process, first filtering out the information of each section that occupies physical memory for the acquired target process memory data, filtering out the information of each section that does not occupy physical memory, and then further judging and filtering the information of the core data content items to be saved according to preset filtering rules to obtain the core data content items to be saved; sequentially integrating and grouping the core data content items to be saved obtained after filtering, and sequentially performing memory application and data compression processing on each grouped data unit, and finally saving the core compressed file. The present invention also provides a corresponding core compressed file generation device.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and more particularly, relates to a method and device for generating a core compressed file. Background Art

[0002] In an embedded Linux system, when a program crashes, a core file (core dump file) is generally generated in a specified directory. Developers use gdb to call the core file to analyze the program exception point. In some abnormal scenarios, multiple core files may be generated continuously, which will cause insufficient system disk storage space, thereby causing system abnormalities and preventing new files from being stored normally. In some extreme cases, such as when the system disk storage space is seriously insufficient and the core file to be generated by the system is too large, the core file cannot be generated normally. The existing core file compression scheme is still based on the generated core file through compression instructions. This still cannot avoid the risk of core generation failure, and the compression rate is relatively low and difficult to control, which cannot meet the requirements for the disk space occupied by the compressed core file. Summary of the Invention

[0003] In view of the limitations of existing core file compression schemes, the present invention provides a core compressed file generation method and device, which can flexibly intercept core data content and directly compress the core original data to save it as a core compressed file, thereby improving the compression rate of the core file and reducing the requirement for disk space size.

[0004] To achieve the above object, according to one aspect of the present invention, a method for generating a core compressed file is provided, comprising the following steps:

[0005] S1. Get the process pid that triggers the coredump process, and query the maps and smaps information through the process pid to obtain the memory usage and distribution of the process;

[0006] S2. Core data is intercepted based on the memory usage and distribution of the process. For the acquired target process memory data, the section information occupying physical memory is first filtered out, and the section data not occupying physical memory is filtered out. Then, the information of the content items to be saved in the core data is further judged and filtered according to the preset filtering rules to obtain the core data content items to be saved.

[0007] S3. The core data content items to be saved after filtering are sequentially integrated and grouped. Each grouped data unit is sequentially memory-required and data compressed, and finally saved as a core compressed file.

[0008] In one embodiment of the present invention, querying maps and smaps information based on the process pid to obtain the memory usage and distribution of the process is specifically performed by viewing the / proc / ${pid} / maps and / proc / ${pid} / smaps information items to obtain the memory data distribution of the process.

[0009] In one embodiment of the present invention, the maps information item is queried to obtain the total section statistics of the process, and then the smaps information item is queried to obtain the virtual memory and physical memory usage of each memory segment.

[0010] In one embodiment of the present invention, the virtual memory and physical memory usage of each memory segment is obtained, specifically: the Size field is the virtual memory size of a memory segment, and the Rss field is the physical memory size of a memory segment.

[0011] In one embodiment of the present invention, the filtering rule is:

[0012] For the listed multiple dynamic library section information, query the permission value of the section, map PROT_NONE, and ignore the memory data of the section.

[0013] In one embodiment of the present invention, the filtering rule is:

[0014] For the stack overflow protection area of ​​each child thread of the listed target process and the section information of the stack space area of ​​the child thread, the memory data of the section is ignored.

[0015] In one embodiment of the present invention, step S3 is specifically as follows:

[0016] Step S31: obtaining the actual remaining physical memory size of the current system as a reference value for large memory application;

[0017] Step S32: sorting the PT_LOAD segments after filtering and intercepting the data from small to large strategy;

[0018] Step S33: After sorting, the PT_LOAD segment data is integrated and grouped according to the strategy to form a plurality of group unit data;

[0019] Step S34: The multiple grouping unit data are sequentially processed through physical memory application and data compression function flow to form the final saved data, which is then written into the core compressed file.

[0020] In one embodiment of the present invention, after each packet data is compressed, a segment identification header needs to be added when each segment of compressed data is saved. The information includes the segment ID number and the segment length.

[0021] In one embodiment of the present invention, obtaining the process PID that triggers the coredump process is specifically:

[0022] When a process runs abnormally, it will receive an abnormal signal from the kernel, thereby triggering the signal processing process. The struct task_struct structure in the processing process is used to obtain the pid of the corresponding process.

[0023] According to another aspect of the present invention, a core compressed file generating device is provided, comprising a data acquisition unit, a data processing unit, and a core compressed file generating unit, wherein:

[0024] The data acquisition unit is used to obtain the process pid that triggers the coredump process, and query maps and smaps information through the process pid to obtain the memory usage and distribution of the process;

[0025] The data processing unit intercepts core data based on the memory usage and distribution of the process. For the acquired target process memory data, the unit first filters out the section information that occupies physical memory, filters out the section data that does not occupy physical memory, and then further determines and filters the information of the core data content items to be saved according to the preset filtering rules to obtain the core data content items to be saved.

[0026] The core compressed file generation unit is used to sequentially integrate and group the core data content items to be saved after filtering, and each grouped data unit is sequentially subjected to memory application and data compression processing, and finally saved as a core compressed file. In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the existing technology:

[0027] (1) According to the technical solution of the present invention, when a system exception generates core data, the valid physical memory data to be saved is first filtered and intercepted, and the saved data items are integrated and grouped. After the data of the multiple grouping units are compressed, they are saved as core compressed files. This can effectively save the disk space occupied by the file in the system directory space, reduce the occurrence of system abnormalities caused by insufficient disk storage space, and thus improve the stability of system operation. The core file content saved after content interception is more targeted and can also simplify problem location and troubleshooting.

[0028] (2) The technical solution of the present invention effectively trims the core file data content and does not generate the core original file. Instead, it directly generates the core compressed file in the kernel state, which greatly alleviates the impact on disk space occupation and improves the stability of system operation. By integrating and grouping the core data content items and compressing them in groups, the compression rate is effectively improved, and the risk of core file generation failure is reduced. The trimmed core file also simplifies the fault problem analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of a method for generating a core compressed file disclosed in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the usage of virtual memory and physical memory of a memory segment in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of multiple dynamic library section information in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of section information of each sub-thread of a target process in an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of the principle of sorting and grouping original core data items in an embodiment of the present invention;

[0034] Figure 6 The present invention discloses a core compressed file generating device. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0036] In view of the limitations of existing core file compression schemes, in order to improve the compression rate of core files and thus reduce the requirements for disk space size, the present invention provides a core compressed file generation scheme that can flexibly intercept core data content and directly perform data compression processing on the core original data to save it as a core compressed file. Usually, the core original data obtained and saved are very large, but a lot of the saved memory data are invalid data content, which is useless for locating fault problems and takes up too much storage space. In response to this situation, relevant filtering strategies can be formulated to intercept the actual saved core data content, and at the same time adjust the order of the original core data saved content, reintegrate the saved original data, and obtain a higher compression rate after being processed by a compression algorithm, thereby reducing the impact of the generated core file on the disk space occupied, thereby improving system stability, and simplifying the location analysis of the problem by cutting the saved core data content.

[0037] To achieve the above purpose, if Figure 1 As shown, an embodiment of the present invention provides a method for generating and managing core compressed files, including:

[0038] Step S1: Get the process pid that triggers the coredump process, and query the maps and smaps information through the process pid to obtain the memory usage and distribution of the process;

[0039] A coredump file is a file that is generated when an application exits abnormally due to various exceptions during operation. Under certain conditions, a core file (core dump file) is generated. The file contains the memory information, register status, stack pointer, and function call stack information during program execution.

[0040] When a process runs abnormally, it will receive an abnormal signal from the kernel, thereby triggering the signal processing process. The struct task_struct structure in the processing process is used to obtain the pid information of the corresponding process.

[0041] Specifically, during the coredump processing at the end of the signal processing flow, you can use the / proc / ${pid} interface to access and query the maps and smaps field information to obtain the memory usage and distribution of the process; specifically, by viewing the / proc / ${pid} / maps and / proc / ${pid} / smaps information items, you can obtain the memory data distribution of the process;

[0042] Specifically, query the maps information item to obtain the total section statistics of the process, and then query the smaps information item to obtain the actual physical memory information occupied by each section; maps refers to the corresponding virtual address information of a process's code segment, stack area, heap area, dynamic library, etc.; smaps displays more detailed memory usage data information for each partition;

[0043] The maps information item mainly counts the memory segment address and size information of the corresponding process, and further querying the smaps information item can obtain the virtual memory and physical memory usage of each memory segment; Figure 2 As shown:

[0044] The Size field is the virtual memory size of a certain memory segment, and the Rss field is the physical memory size of a certain memory segment. According to the Rss field information, when data is intercepted, the sections that do not occupy physical memory are filtered out (i.e., the Rss value is 0). At the same time, for the saved section data information, only the section that occupies physical memory is retained (i.e., the data size is the Rss value). The existing core file solution saves all virtual memory data, but the actual situation is that when using core file data to restore and locate fault information, the useful information is only the data information currently saved in the physical memory. Based on this, the physical memory item information of each memory segment is first analyzed, and the core data content is preliminarily intercepted. Only the physical memory item size of each memory segment is saved (the size corresponding to Rss), thereby filtering out the section data items that do not occupy physical memory.

[0045] Step S2: Core data is intercepted based on the memory usage and distribution of the process. For the acquired target process memory data, the section information occupying physical memory is first filtered out, and the section data not occupying physical memory is filtered out. Then, the information of the content items to be saved in the core data is further judged and filtered according to the preset filtering rules to obtain the core data content items to be saved.

[0046] The present invention has formulated the following memory data filtering rules to further filter the saved section data, with the aim of filtering out invalid section memory data items (i.e., the memory segment data content has no effect on fault restoration and location and can be omitted from the core file);

[0047] Rule 1: If Figure 3 As shown, for the multiple dynamic library section information listed, query the permission value of the section, and for PROT_NONE mapping (that is, ---p permission item), the memory data of the section can be ignored;

[0048] Rule 2: If Figure 4 As shown, for the section information of each child thread of the listed target process (stack overflow protection area and child thread stack space area), the contents of these two sections are private to each child thread. The data content does not affect the fault analysis and location, so the memory data of this section is ignored (the figure shows that the process has section data of three child threads. For all child threads of a specific process, the data contents of these two sections do not need to be saved);

[0049] By using the above two filtering conditions, the data saved in the core file can be further reduced;

[0050] The memory data screening rules in the present invention include but are not limited to the above rules. The data items to be saved can be further screened according to the core data content that actually needs to be saved, such as data screening by memory data types that meet different situations (such as anonymous memory, shared memory, large page data memory, etc.);

[0051] It should be noted that the core file data structure mainly consists of ELF Header, Program Headers, PT_NOTEsegment, and PT_LOAD segment. The data of the first three structure segments are not in the memory and can be obtained and saved according to the existing interface. The data items processed by the above method refer to the PT_LOAD segment data.

[0052] Step S3: The core data content items to be saved after filtering are sequentially integrated and grouped, and each grouped data unit is sequentially memory-required and data-compressed, and finally saved as a core compressed file.

[0053] The reason for order integration and grouping here is that the core data memory items are usually very large. In kernel processing, physical memory cannot be applied for all core data items at once. Memory application, storage and data compression processing need to be performed based on the filtered section data content. Order integration and grouping processing will first sort the saved section data content. Multiple section data can be integrated into one group of data. The purpose is to reduce data grouping items. Then, data compression and storage processing are performed in each group to improve the overall data compression rate.

[0054] In the process of the present invention, it is taken into consideration that the core original data cannot be saved in memory at one time (physical memory is limited), and if each PT_LOAD segment data is saved in memory once and then compressed, the data processing efficiency is low and the overall data compression size is difficult to control. Based on this situation, the present invention gives priority to sorting the PT_LOAD segment data in order of content size, reorganizing the storage order of the PT_LOAD segment data, and strategically integrating and grouping the sorted PT_LOAD segment data into multiple group unit data, so that the size of the combined unit data is as large as possible and less than the threshold of the remaining physical memory (free_mem). In this way, after compression processing of each unit data, the overall compression efficiency of the core data will be greatly improved, so that the saved core compressed file will be as small as possible.

[0055] Therefore, the core data content items can be obtained through the first two steps. The core data content items after filtering and interception have been greatly reduced. Based on this data content item, integration and compression are continued, specifically:

[0056] Step S31: Obtain the actual remaining physical memory size (free_mem) of the current system as a reference value for large memory application;

[0057] Step S32: sorting the PT_LOAD segments after filtering and intercepting the data from small to large strategy;

[0058] Step S33: After sorting, the PT_LOAD segment data is integrated and grouped according to the strategy to form a plurality of group unit data;

[0059] Step S34: multiple grouping unit data are sequentially processed through physical memory application and data compression function processes to form the final saved data, which is then written into the core compressed file;

[0060] like Figure 5 As shown in the figure, the original core data items are sorted in ascending order, and then integrated and grouped into multiple grouping units, so that the content size of each unit is smaller than the physical memory threshold (free_mem); after the data compression process, the overall compression efficiency is higher;

[0061] After compressing each packet data, a segment identification header must be added when saving each compressed data segment. The information includes the segment ID number and segment length. The purpose is to parse the segment identification header during decompression to restore the core original data for fault location analysis.

[0062] Figure 6A schematic diagram of the structure of a core compressed file generating device provided by an embodiment of the present invention, comprising a data acquisition unit, a data processing unit, and a core compressed file generating unit;

[0063] The data acquisition unit is used to obtain the process pid that triggers the coredump process, and query maps and smaps information through the process pid to obtain the memory usage and distribution of the process;

[0064] The data processing unit intercepts core data based on the memory usage and distribution of the process. For the acquired target process memory data, the unit first filters out the section information that occupies physical memory, filters out the section data that does not occupy physical memory, and then further determines and filters the information of the core data content items to be saved according to the preset filtering rules to obtain the core data content items to be saved.

[0065] The core compressed file generating unit is used to sequentially integrate and group the core data content items to be saved after filtering, and each grouped data unit is sequentially subjected to memory application and data compression processing, and finally saved as a core compressed file.

[0066] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a core compressed file, characterized in that: The steps include: S1. Get the process pid that triggers the coredump process, and query the maps and smaps information through the process pid to obtain the memory usage and distribution of the process; S2. Core data is intercepted based on the memory usage and distribution of the process. For the acquired target process memory data, the section information occupying physical memory is first filtered out, and the section data not occupying physical memory is filtered out. Then, the information of the content items to be saved in the core data is further judged and filtered according to the preset filtering rules to obtain the core data content items to be saved. S3. The core data content items to be saved after filtering are sequentially integrated and grouped, and each data unit after grouping is sequentially subjected to memory application and data compression processing, and finally saved as a core compressed file; the step S3 is specifically as follows: step S31: obtaining the actual remaining physical memory size of the current system as a reference value for large memory application; step S32: firstly perform a small to large strategy integration and sorting on each PT_LOAD segment data after data filtering and interception; step S33: after sorting, each PT_LOAD segment data is segmented and grouped according to the strategy to form a plurality of group unit data; step S34: multiple group unit data are sequentially subjected to physical memory application and data compression function flow to form the final saved data, and then written into the core compressed file.

2. The core compressed file generation method according to claim 1, wherein: The memory usage and distribution of the process are obtained by querying the maps and smaps information through the process pid, specifically: obtaining the memory data distribution of the process by viewing the / proc / ${pid} / maps and / proc / ${pid} / smaps information items.

3. The core compressed file generation method according to claim 2, wherein: Query the maps information item to obtain the total section statistics of the process, and then query the smaps information item to obtain the virtual memory and physical memory usage of each memory segment.

4. The method for generating a core compressed file according to claim 3, wherein: Get the virtual memory and physical memory usage of each memory segment. Specifically, the Size field is the virtual memory size of a memory segment, and the Rss field is the physical memory size of a memory segment.

5. The method for generating a core compressed file according to claim 1 or 2, wherein: The filtering rules are: For the listed multiple dynamic library section information, query the permission value of the section, map PROT_NONE, and ignore the memory data of the section.

6. The method for generating a core compressed file according to claim 1 or 2, wherein: The filtering rules are: For the stack overflow protection area of ​​each child thread of the listed target process and the section information of the stack space area of ​​the child thread, the memory data of the section is ignored.

7. The method for generating a core compressed file according to claim 1, wherein: After compressing each packet data, a segment identification header needs to be added when saving each compressed data segment, and the information includes the segment ID number and segment length.

8. The method for generating a core compressed file according to claim 1, wherein: The process pid that triggers the coredump process is obtained as follows: When a process runs abnormally, it will receive an abnormal signal from the kernel, thereby triggering the signal processing process. The struct task_struct structure in the processing process is used to obtain the pid of the corresponding process.

9. A core compressed file generating device, characterized in that: It includes a data acquisition unit, a data processing unit, and a core compressed file generation unit, wherein: The data acquisition unit is used to obtain the process pid that triggers the coredump process, and query maps and smaps information through the process pid to obtain the memory usage and distribution of the process; The data processing unit intercepts core data based on the memory usage and distribution of the process. For the acquired target process memory data, the unit first filters out the section information that occupies physical memory, filters out the section data that does not occupy physical memory, and then further determines and filters the information of the core data content items to be saved according to the preset filtering rules to obtain the core data content items to be saved. The core compressed file generation unit is used to sequentially integrate and group the core data content items to be saved after filtering, and each data unit after grouping is sequentially subjected to memory application and data compression processing, and finally saved as a core compressed file; specifically: the actual remaining physical memory size of the current system is obtained as a reference value for large memory application; each PT_LOAD segment data after data filtering and interception is first integrated and sorted from small to large according to the strategy; each PT_LOAD segment data after sorting is integrated and grouped according to the strategy to form multiple group unit data; multiple group unit data are sequentially subjected to physical memory application and data compression function process to form the final saved data, and then written into the core compressed file.

Images