Memory duplicate removal method and device based on page similarity

Abstract

The invention discloses a memory deduplication method and device based on page similarity, and the device comprises two components, and is used for sensing page screening and fine-grained duplicated data deletion. Application perception page screening is responsible for screening out anonymous pages with high similarity in the memory occupied by each application, fine-grained duplicated data deletion is responsible for splitting the screened pages into blocks, and finer-grained memory duplicated data deletion is realized by maintaining a block content table, a block hash table and a page address table in the memory. And the page recovery operation is realized before page access. According to the method, codes of application programs do not need to be modified, the method is easily integrated in a Linux kernel, and compared with a page sharing method, more available memory space is provided. The result of experimental evaluation by using real equipment shows that 30.45% of memory can be saved averagely, and 30.51% of application program caching capability is improved.

Description

Technical Field

[0001] The present invention relates to the field of application flash memory cache technology, and in particular to a memory deduplication method and device based on page similarity. Background Art

[0002] In modern mobile systems, memory is usually divided into pages, which are generally divided into file pages and anonymous pages. Since file pages can be written back or released, their repetition rate is relatively low, and the proportion of memory occupied is not large; while anonymous pages usually account for a higher proportion in the system. Although anonymous pages can be compressed through ZRAM, mobile users often do not actively close an application when they are not in use, but switch it to the background, resulting in a continuous lack of available memory. At the same time, some applications will be killed by the system due to insufficient free memory space, which affects the user experience.

[0003] When the available memory is insufficient, users will noticeably feel that the speed of opening new applications slows down. The reason is that new applications require a lot of memory to start, and the system must first reclaim the memory occupied by some applications before allocating enough space for new applications. This recycling process is relatively time-consuming. The kswapd process in the kernel space is mainly responsible for reclaiming file pages, because most file pages are not dirty pages and can be released directly. In contrast, a large number of anonymous pages are allocated during the use of applications (not at the startup stage). Even after compression with ZRAM, they still occupy a certain amount of memory space, and over time they become the main source of memory consumption. This also means that if you want to optimize memory usage, you need to focus on how to effectively delete redundant data between anonymous pages.

[0004] Most existing redundant data removal schemes are based on content-based page sharing (such as KSM), which removes duplicates in units of whole pages: only when two pages are exactly the same will they be merged, otherwise they will not be processed. However, the number of identical pages in mobile systems is limited, and this whole-page level deduplication method is difficult to fully utilize the partial similarities between pages, and it is easy to waste space that can be deduplicated. Android applications written in Java and managing resources in units of objects will generate a large number of similar but not identical anonymous pages during operation, which makes it possible to achieve redundant data removal at a finer granularity. Summary of the invention

[0005] In order to overcome the problems existing in the above technologies, the purpose of the present invention is to provide a memory deduplication method and device based on page similarity. The basic idea is to identify the same parts in two similar pages and deduplicate them, which consists of two key parts: application-aware page filtering and fine-grained deduplication.

[0006] The specific technical solution for achieving the purpose of the present invention is: A memory deduplication method based on page similarity includes the following steps: Step 1: Select the non-sparse anonymous pages of the background inactive applications as the deduplication objects; specifically: identify the background inactive applications through the adj value in the Android system, and filter out the non-sparse anonymous pages therein; Step 2: Split the screened non-sparse anonymous pages into multiple blocks, sample the blocks and calculate hash values ​​to obtain their fingerprints; the hash algorithm used is lightweight MD5; as long as one of the fingerprints in the sampled blocks of two pages is the same, then the two non-sparse anonymous pages are similar; Step 3: Perform block-level deduplication on similar pages. For each block obtained by splitting similar pages, check whether its hash value already exists in the block hash table. If not, write the block data to the Union part of the block content table, and write the hash value of the block and the address of the block in the Union part of the block content table to the block hash table. If it exists, compare the block data with the block data corresponding to the existing hash value in the block hash table byte by byte to avoid hash conflicts. If there is a conflict, it means that the block data is not repeated, and the data of the current block is directly written to the Union part of the block content table without updating the block hash table. If there is no conflict, it means that the two blocks of data are repeated, and the address of the existing repeated block is written to the Union part of the block content table, and the Tag part of the block content table is set to 1, indicating that the data corresponding to this bit is metadata rather than the original block content. Step 4: Repeat step 3 to the last block of this page, and then start to do the same operation for each block of the next similar page until all pages of the background inactive application have been deduplicated; Step 5: For non-sparse anonymous pages that are split into blocks, recovery is completed before they are accessed, making block-level deduplication compatible with Linux's memory paging mechanism: During the deduplication process, the mapping information of pages and blocks is recorded; if the candidate block being scanned is the first block of a similar page, the virtual page number of the page and the address of the block in the Union part of the block content table are written into the page address table; when the page needs to be accessed, different recovery processes are performed according to two different situations, as follows: a. Page recovery during application caching: When the application is in the background and has been deduplicated, if the system needs to access a split page, the page is recovered according to the page address table and block content table. After the page is recovered, the corresponding entry in the page address table is deleted, but no changes are made to the block content table because the related block data may still be used by other pages. These recovered pages are no longer split and deduplicated. b. Page recovery during application switching: When an application that has been deduplicated switches to the foreground, two operations, fast recovery and slow recovery, are performed on the memory pages of the application; fast recovery means that the page to be accessed has been split, and the page is recovered according to the page address table and block content table; slow recovery means that the page recovery operation is performed sequentially for pages that have not been accessed or recovered according to the content in the page address table; when computing resources are sufficient, the slow recovery operation is performed to ensure user experience; in other cases, the fast recovery is performed; Step 6: When the page is restored, if the Tag part of the block content table corresponding to the page is 0, the data in the Union part of the block content table is directly read and reassembled into a page; if the Tag part of the corresponding block content table is 1, the address of the duplicate block is obtained according to the content of the Union part of the block content table, and the block data of the corresponding address is found in the block content table, and the data is spliced ​​and restored to the page; because only the first duplicate block will be spliced ​​when the page of the duplicate block is restored, memory deduplication is completed.

[0007] Furthermore, the block content table consists of two parts: Tag and Union; Tag is a bitmap used to identify whether the content corresponding to the bit is original memory block data or metadata information (an integer representing a repeated block address); Union is a continuous virtual memory space used to store block data or metadata (representing a repeated block address).

[0008] Furthermore, the block hash table is used to check whether the hash value of each block of a similar page already exists: The block hash table is responsible for storing the hash value of the memory block and the address of the memory block in the Union part of the block content table during the deduplication process; after the deduplication is completed, the block hash table space is released.

[0009] Furthermore, the page address table is responsible for locating the first block address of a page in the Union part of the block content table.

[0010] A memory deduplication device based on page similarity, used to implement the above-mentioned memory deduplication method based on page similarity, comprising: Application-aware page filtering module: monitors the status of applications and identifies inactive applications; identifies anonymous pages belonging to the application and calculates the fingerprints of these anonymous pages; only anonymous pages with high similarity obtained through fingerprint comparison will be further processed; Fine-grained deduplication module: splits the filtered pages into blocks and performs block deduplication. When a deduplicated application is used again, the split pages are reorganized and restored. The two modules work in a daemon process. When the application is switched to the background and is considered inactive, the daemon process is awakened and the two modules start working.

[0011] Compared with the prior art, the method proposed in the present invention effectively reduces the total memory consumption, improves the application cache capability, and significantly improves the system performance and user experience. The evaluation results show that the present invention reduces the total memory consumption by 30.45% on average and improves the application cache capability by 30.51%. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is an example diagram of the internal mechanism of fine-grained data deduplication; Figure 3 It is the working flow chart of the device of the present invention. DETAILED DESCRIPTION

[0013] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0014] The memory deduplication method based on page similarity of the present invention is to identify anonymous pages belonging to the application by performing application-aware page screening on inactive applications in the background and calculate the fingerprints of these anonymous pages. Only anonymous pages with high similarity obtained through fingerprint comparison will be further processed. Fine-grained deduplication splits the screened pages into blocks and performs block deduplication. When a deduplicated application is used again, the split pages will be reorganized and restored.

[0015] The process of application-aware page screening is as follows: First, in order to save space by sacrificing a small amount of memory to significantly reduce CPU and memory overhead, the present invention screens out non-sparse pages from the collected pages. Second, each page is split into multiple blocks to identify more redundant data. The present invention samples these blocks and calculates their hash values ​​to obtain their fingerprints. According to the principle of spatial locality, if two pages have two identical memory blocks, there is a high probability that there are other identical blocks. As long as one of the fingerprints in the sampled blocks of two pages is the same, the two pages are similar.

[0016] In order to achieve fine-grained deduplication, solve the problems of how to organize data after deduplication, how to find the same data blocks, and how to quickly restore split pages, this paper invents and designs three data structures: (1) Block Content Table (BC-Table) is used to store memory block data. It consists of two parts: Tag and Union. Tag is a bitmap that is used to identify whether the content corresponding to the bit is the original memory block data or metadata information (an integer representing the address of the duplicate block). Union is a continuous virtual memory space used to store block data or metadata. (2) Block Hash Table (BH-Table) is responsible for storing the hash values ​​of memory blocks and their addresses in the Union part of BC-Table during the deduplication process. After deduplication is completed, the BH-Table space can be released. (3) Page Address Table (PA-Table) is responsible for locating the first block address of a page in the Union part of BC-Table.

[0017] The basic process of fine-grained deduplication is as follows: For each block filtered out by the application-aware page, search the BH-Table based on its hash value to check whether the same hash value already exists. If the same hash value does not exist, write the block data to the Union part of the BC-Table (abbreviated as BC-Table_Union), and write the hash value of the block and the address of the block in the BC-Union to the BH-Table. If there is a hash value that is the same as the block, compare the block data with the block data corresponding to the existing hash value in the BH-Table byte by byte to avoid different data generating the same hash value, that is, hash collision. If the data of the two blocks are not the same, write the data of the current block directly to the BC-Table_Union without updating the BH-Table. If the data of the two blocks are indeed the same, write the address of the existing same block to the BC-Table_Union, and set the corresponding BC-Table_Tag to 1, indicating that the data corresponding to this bit is metadata rather than the original block content. Before accessing a split page, a page recovery operation is performed: if the candidate block being scanned is the first block of a page, the virtual page number of the page and the address of the block in BC-Table_Union are written into the PA-Table.

[0018] The page recovery operation can be specifically divided into two scenarios: (1) Page recovery during application caching. When the application is in the background and has been deduplicated, if the system needs to access a split page, it can recover the page based on the PA-Table and BC-Table. After the page is recovered, the corresponding table entry in the PA-Table is deleted, but no changes are made to the BC-Table because the related block data may still be used by other pages. In addition, these recovered pages do not need to be split and deduplicated again. (2) Page recovery during application switching. If the deduplicated application is switched to the foreground, two operations, fast recovery and slow recovery, will be performed on the memory pages of the application. Fast recovery means that if the page to be accessed has been split, the page is recovered based on the PA-Table and BC-Table. Slow recovery means that according to the content in the PA-Table, the page recovery operation is performed sequentially for the pages that have not been accessed or recovered. When computing resources are sufficient, this slow recovery operation can be performed to ensure user experience. Example

[0019] like Figure 3 As shown in the figure, it is a flow chart of the memory deduplication device based on page similarity. Two components are designed: application-aware page filtering and fine-grained deduplication. In general, application-aware page filtering uses a page sampling method to filter out similar pages of inactive applications, greatly reducing the overhead of hash calculation and page comparison. Fine-grained deduplication splits pages into blocks to perform redundant data removal, and maintains three tables, BC-Table, BH-Table and PA-Table, to store deduplicated data and metadata information, and performs page recovery operations before the application becomes an active application when the system accesses the split page.

[0020] The process of the present invention is shown in Figure 1 . It is mainly implemented in the following steps: 1) Split each non-sparse page of an inactive application into multiple blocks to identify more redundant data. If the block size is 2 order B, the page size is in bytes, then each page can be divided into blocks. In mobile systems, a page is usually 4KB. If the parameter order is set to 6, the block size is 64B, and a page can be split into 64 blocks. Assuming that a page has N={S1, S2, ..., Sn} sampling blocks, the fingerprint of the page can be obtained by calculating the hash values ​​of these N blocks. The present invention uses MD5, a lightweight hash algorithm, to calculate the hash value.

[0021] 2) For each block, check if the same hash value already exists in the BH-Table. Figure 2 , when the block with content A is scanned again, the hash value of the block is calculated to be h0. Searching the BH-Table finds that the hash value h0 already exists in the table, and the corresponding address is 0, so the data at address 0 in the BC-Table is compared byte by byte with the data of the currently scanned block. Since the contents of the two blocks are the same, address 0 is written into the BC-Table_Union, that is, the data corresponding to address 2048, and the corresponding tag is set to 1. When the memory block with content F is scanned, although the same hash value already exists in the BH-Table, the contents of the two blocks are actually different, so the original content of the block is written into the BC-Table_Union, and the BH-Table is not updated. In this way, by sacrificing a small amount of memory savings, the memory comparison overhead caused by hash conflicts can be significantly reduced.

[0022] 3) When accessing page data, the virtual address must first be converted to a physical address according to the page table in order to obtain the correct data. However, the block-level deduplication method conflicts with this mechanism because the method splits the page into blocks instead of using the page as the basic unit. To solve this problem, the present invention performs a page recovery operation before accessing the split page. Therefore, the mapping information of the page and the block needs to be recorded. Figure 2 For example, the first blocks of page 1 and page 3 have addresses 0 and 3076 in BC-Table_Union respectively, so (1, 0) and (3,3076) are recorded in PA-Table.

[0023] 4) Perform page recovery operations such as Figure 2 , if you want to restore page 3, according to PA-Table, the data of the first block of page 3 needs to be read from the 3076th byte of BC-Table_Union. Since the BC-Table_Tag corresponding to this block is 0, this means that the data stored in this block is the original memory block data, so it can be read directly. The same operation is performed on the contents of the following two blocks. Since the BCTable_Tag corresponding to the data starting from the 6146th byte is 1, it means that the data starting from this byte is metadata. According to the content of BC-Table_Union, the original block data needs to be read from the 2052th byte, that is, memory block C. After the read data is spliced ​​in sequence, the recovery operation of the page is completed.

[0024] In addition, this implementation requires two types of overhead: storage and performance overhead. Storage overhead includes mapping tables that need to be maintained, such as PA-Table and BC-Table. The present invention effectively screens pages before page splitting, so many pages do not need to be recorded in the table. Secondly, a simple method is used to design the structure of the mapping table. Therefore, the storage overhead cost is very small and can be ignored. The performance overhead comes from the number of hash calculations and comparisons. Since only pages belonging to the same application need to be compared, and a faster and more efficient small fingerprint calculation (µs level) is used, the process overhead can be ignored.

Claims

1. A memory deduplication method based on page similarity, characterized in that: The following steps are involved: Step 1: Select the non-sparse anonymous pages of the background inactive applications as the deduplication objects; specifically: identify the background inactive applications through the adj value in the Android system, and filter out the non-sparse anonymous pages therein; Step 2: Split the screened non-sparse anonymous pages into multiple blocks, sample the blocks and calculate the hash values ​​to obtain their fingerprints; the hash algorithm used is lightweight MD5; as long as one of the fingerprints in the sampled blocks of two pages is the same, the two non-sparse anonymous pages are similar; Step 3: Perform block-level deduplication on similar pages. For each block obtained by splitting similar pages, check whether its hash value already exists in the block hash table. If it does not exist, write the block data to the Union part of the block content table, and write the hash value of the block and the address of the block in the Union part of the block content table to the block hash table; If it exists, the block data is compared byte by byte with the block data corresponding to the existing hash value in the block hash table to avoid hash conflicts; If there is a conflict, it means that the block data is not repeated, and the data of the current block is directly written into the Union part of the block content table without updating the block hash table; if there is no conflict, it means that the two blocks of data are repeated, and the address of the existing repeated block is written into the Union part of the block content table, and the Tag part of the block content table is set to 1, indicating that the data corresponding to this bit is metadata rather than the original block content; Step 4: Repeat step 3 to the last block of this page, and then start to do the same operation for each block of the next similar page until all pages of the background inactive application have been deduplicated; Step 5: For non-sparse anonymous pages that are split into blocks, recovery is completed before they are accessed, making block-level deduplication compatible with Linux's memory paging mechanism: During the deduplication process, the mapping information of pages and blocks is recorded; if the candidate block being scanned is the first block of a similar page, the virtual page number of the page and the address of the block in the Union part of the block content table are written into the page address table; when the page needs to be accessed, different recovery processes are performed according to two different situations, as follows: a. Page recovery during application caching: When the application is in the background and has been deduplicated, if the system needs to access a split page, the page is recovered according to the page address table and block content table. After the page is recovered, the corresponding entry in the page address table is deleted, but no changes are made to the block content table because the related block data may still be used by other pages. These recovered pages are no longer split and deduplicated. b. Page recovery during application switching: When an application that has been deduplicated switches to the foreground, two operations, fast recovery and slow recovery, are performed on the memory pages of the application; fast recovery means that the page to be accessed has been split, and the page is recovered according to the page address table and block content table; slow recovery means that the page recovery operation is performed sequentially for pages that have not been accessed or recovered according to the content in the page address table; when computing resources are sufficient, the slow recovery operation is performed to ensure user experience; in other cases, the fast recovery is performed; Step 6: When the page is restored, if the Tag part of the block content table corresponding to the page is 0, the data in the Union part of the block content table is directly read and reassembled into a page; if the Tag part of the corresponding block content table is 1, the address of the duplicate block is obtained according to the content of the Union part of the block content table, and the block data of the corresponding address is found in the block content table, and the data is spliced ​​and restored to the page; because only the first duplicate block will be spliced ​​when the page of the duplicate block is restored, memory deduplication is completed.

2. A memory deduplication method based on page similarity as claimed in claim 1, characterized in that: The block content table is composed of Tag and Union; Tag is a bitmap used to identify whether the content corresponding to the bit is original memory block data or metadata information, that is, an integer representing a repeated block address; Union is a continuous virtual memory space used to store block data or metadata, that is, a repeated block address.

3. A memory deduplication method based on page similarity as claimed in claim 1, characterized in that: The block hash table is used to check whether the hash value of each block of a similar page already exists: The block hash table is responsible for storing the hash value of the memory block and the address of the memory block in the Union part of the block content table during the deduplication process; after the deduplication is completed, the block hash table space is released.

4. A memory deduplication method based on page similarity as claimed in claim 1, characterized in that: The page address table is responsible for locating the first block address of a page in the Union part of the block content table.

5. A memory deduplication device based on page similarity, used to implement any one of the memory deduplication methods based on page similarity in claims 1 to 4, characterized in that: include: Application-aware page filtering module: monitors the status of applications and identifies inactive applications; Identify anonymous pages belonging to the application and calculate fingerprints of these anonymous pages; Only anonymous pages with high similarity obtained through fingerprint comparison will be further processed; Fine-grained deduplication module: splits the filtered pages into blocks and performs block deduplication. When a deduplicated application is used again, the split pages are reorganized and restored. The two modules work in a daemon process. When the application is switched to the background and is considered inactive, the daemon process is awakened and the two modules start working.

Images