A method for supporting local data caching and high-concurrency secure access

Abstract

The present invention belongs to the technical field of content distribution networks, specifically a method for supporting local data caching and highly concurrent secure access. The method involves creating a shared memory block using the mmap function provided by the operating system. The shared memory is mapped to a specific file on disk. When data is written to the shared memory, the operating system synchronizes the shared memory data with the file in real time. The method supports highly concurrent secure reading and writing of hundreds of millions of data points, while maintaining low memory usage and zero memory fragmentation. Cached data is prevented from being lost when the CDN cache gateway process exits normally or abnormally. When the data volume reaches the cache storage limit, the total cached data size is controlled, automatically eliminating historical data.

Description

Technical Field

[0001] The present invention belongs to the technical field of content distribution networks, and in particular relates to a method for supporting local data caching and high-concurrency secure access. Background Art

[0002] Content Delivery Network (CDN) is a new type of network content service system built on IP network. It provides content distribution and services based on the efficiency requirements, quality requirements and content order of content access and application. CDN is an intelligent virtual network built on the existing network foundation. It relies on edge servers deployed in various places and uses the load balancing, content distribution, scheduling and other functional modules of the central platform to enable users to obtain the required content nearby, reduce network congestion and improve user access response speed.

[0003] Problems with existing technologies:

[0004] Currently, CDN cache gateways often involve querying whether a certain element is in a dataset. For example, querying whether the client IP address is in an IP blacklist database or querying whether the URL requested by the client is in a URL ban database. While theoretically feasible, if the IP blacklist and URL ban databases are stored in one place, such as a database, and then queried through the CDN node machine, this approach does not work in practice for the following reasons:

[0005] 1. There are tens of thousands of CDN node machines. Assuming that the QPS (query per second) of a node machine is 3000, and all node machines access the database, no single database host can meet the search query requirements of 30 million QPS visits. If a database cluster is built, a large number of database clusters will be required, and the management, maintenance, and operating costs will be very high.

[0006] 2. CDN cache gateway performance is of paramount importance. After all, the shorter the time it takes for a user to open a web page or video, the better. If each request accesses the database through the network protocol stack, network fluctuations will cause the request time to be longer. On the other hand, it depends on the high availability of the database. If the service goes down, it will affect the entire request.

[0007] Since the remote access method mentioned above is difficult to implement, delegating data set access to each CDN node for local access is another approach. Summary of the Invention

[0008] The purpose of the present invention is to provide a method that supports local data caching and high-concurrency secure access, which can support high-concurrency secure reading and writing of hundreds of millions of data, has low memory usage, produces zero memory fragmentation, and will not cause cached data loss when the CDN cache gateway process exits normally (abnormally). When the data volume reaches the cache storage limit, it can control the total size of cached data and automatically eliminate historical data.

[0009] The technical solutions adopted by the present invention are as follows:

[0010] A method for supporting local data caching and high-concurrency secure access includes creating a shared memory through an mmap function provided by an operating system, wherein a mapping relationship is established between the shared memory and a corresponding specific file on a disk file.

[0011] When data is written to the shared memory, the operating system synchronizes the shared memory data to the disk file in real time. The shared memory can realize the write cache data process and query data process between the disk file. When the cache data exceeds the shared memory capacity, the LRU elimination algorithm automatically deletes the old data and then writes the new data.

[0012] Among them, the shared memory is divided into a lock management area, a data area, a hash bucket area, an algorithm management area, and a transaction management area. The lock management area is used to control the modification of the critical area data of the hash table to ensure that only one process can modify the critical area data at the same time. When reading and writing data, a shared lock is added through the lock management area. The transaction management area is used to ensure whether the modification is executed or not to avoid the generation of intermediate states. When the process exits abnormally, it is responsible for clearing unreleased locks and rolling back intermediate state transactions.

[0013] The process of writing cache data includes:

[0014] Step 1: Add a write lock in the lock management area, start a transaction in the transaction management area, and temporarily cancel the write protection for the hash bucket memory page;

[0015] Step 2: Calculate the hash value of the cached data according to the hash function;

[0016] Step 3: Check the corresponding hash bucket based on the hash value to see if the bucket has been written. If it has been written, go to the next step; otherwise, go to step 5.

[0017] Step 4: Check the value recorded in the hash bucket to see if the memory data is consistent with the written cache data. If they are consistent, update the data directly and go to step 8.

[0018] Otherwise, continue searching. If no matching data is found, go to the next step.

[0019] Step 5. Find free space in the data area. If there is no free space, go to the next step. Otherwise, go to step 7.

[0020] Step 6: Eliminate the oldest historical record from the algorithm management area and put it into the free space of the data area;

[0021] Step 7: Request memory from the free space in the data area and write the cached data. Store the cached data in the hash bucket at the index position in the data area:

[0022] Step 8: End the transaction in the transaction management area, release the write lock in the lock management area, and restore the write protection of the hash bucket memory page.

[0023] The data query process includes:

[0024] S1. Add a read lock to the lock management area;

[0025] S2. Calculate the hash value of the cached data based on the hash function;

[0026] S3. Check the corresponding hash bucket based on the hash value to see if there is a value in the bucket. If there is, proceed to the next step. Otherwise, return "no match" and release the read lock in the lock management area.

[0027] S4. Check the value in the hash bucket to see if the in-memory data in the data area is consistent with the cached data. If so, proceed to the next step.

[0028] Otherwise, continue searching, and if no matching data is found, return "unmatched" and release the read lock in the lock management area;

[0029] S5. Update data access time in the algorithm management area;

[0030] S6. Return the matching result and release the read lock in the lock management area.

[0031] The technical effects achieved by the present invention are:

[0032] The present invention can achieve high-concurrency and secure reading and writing of hundreds of millions of data with low memory usage. When memory fragmentation occurs, cached data will not be lost when the CDN cache gateway process exits normally (abnormally). When the data volume reaches the cache storage limit, the total size of the cached data can be controlled and historical data can be automatically eliminated.

[0033] The present invention supports high concurrent reading and writing, adds lock control to critical section data, and refines the lock granularity into read lock and write lock, which can increase the reading concurrency.

[0034] The present invention strengthens the security of cached data by adding a memory page protection mechanism, preventing wild pointers or buffer overflow from dirtying shared memory cached data.

[0035] The present invention supports an automatic transaction rollback mechanism, thereby avoiding the occurrence of intermediate transactions and dirty data, and maintaining strong data consistency.

[0036] The present invention avoids memory fragmentation, and the stored data are all of fixed-length type after processing, thus avoiding memory fragmentation caused by different data sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a structural diagram of the present invention;

[0038] Figure 2 This is a flow chart of writing cache data in the present invention;

[0039] Figure 3 This is a flow chart of reading cache data in the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is described in detail below with reference to the following examples. It should be understood that the following text is only used to describe one or more specific embodiments of the present invention and does not strictly limit the scope of protection of the present invention.

[0041] like Figure 1 As shown, a method for supporting local data caching and high-concurrency secure access includes creating a shared memory through the mmap function provided by the operating system, wherein the shared memory establishes a mapping relationship with a corresponding specific file on the disk file.

[0042] When data is written to the shared memory, the operating system synchronizes the shared memory data to the disk file in real time. The shared memory can realize a write cache data flow and a query data flow with the disk file.

[0043] When the cached data exceeds the shared memory capacity, the LRU elimination algorithm automatically deletes the old data before writing the new data. By creating different functional partitions in the shared memory, the shared memory is divided into a lock management area, a data area, a hash bucket area, an algorithm management area, and a transaction management area. This allows the cached data to support high concurrency while meeting high-speed query and insertion requirements. Transaction processing is added to cached data writes, and support is provided for rolling back abnormal mid-transactions. At the same time, security reinforcement is added to the cached data to prevent illegal modifications that could cause cached data to become corrupted. The shared memory is divided into the following areas:

[0044] Lock management area: controls the modification of data in the critical section of the hash table to ensure that only one process can modify the critical section data at the same time.

[0045] The data area is a fixed-length array where each element is of equal size. The elements store specific externally written data and metadata, such as the actual length of the data's query key and the array index of the next data element with the same hash value.

[0046] Hash buckets: These are a row of consecutive buckets, each storing the index of the data in the data area above. Because they only store array indexes, they take up very little space. Properly expanding the hash buckets doesn't take up much memory, but it can reduce the probability of hash collisions after hashing data.

[0047] Algorithm management area: records the order in which each data is accessed. When the memory is full of data, the earliest written but unaccessed data is eliminated.

[0048] Transaction Management Area: This area ensures that all modifications are made or not made, avoiding any intermediate states. It is responsible for clearing any unreleased locks and rolling back intermediate transactions when a process exits abnormally.

[0049] Please refer to Figure 2 As shown, the shared memory write cache data process:

[0050] Shared memory pages are write-protected during initialization to prevent accidental changes.

[0051] 1. Add a write lock in the lock management area, start a transaction in the transaction management area, and temporarily cancel the write protection for the hash bucket memory page.

[0052] 2. Calculate the hash value of the cached data based on the hash function.

[0053] 3. According to the hash value, go to the corresponding hash bucket to check whether the bucket has been written. If it has been written, go to the next step; otherwise, go to step 5.

[0054] 4. Check the data area based on the value recorded in the hash bucket to see if the memory data is consistent with the written cache data. If they are consistent, update directly and go to step 8. Otherwise, continue searching. If no matching data is found, go to the next step.

[0055] 5. Search for free space in the data area. If there is no free space, go to the next step; otherwise, go to step 7.

[0056] 6. Eliminate the oldest historical records from the algorithm management area and put them into the free space in the data area.

[0057] 7. Apply for memory from the free space in the data area and write the cached data, and store the cached data in the hash bucket at the index position of the data area.

[0058] 8. Go to the transaction management area to end the transaction, go to the lock management area to release the write lock, and restore the write protection of the hash bucket memory page.

[0059] Please refer to Figure 3 As shown, query the shared memory data process:

[0060] 1. Add a read lock to the lock management area;

[0061] 2. Calculate the hash value of the cached data based on the hash function;

[0062] 3. According to the hash value, go to the corresponding hash bucket to check whether there is a value in the bucket. If there is, go to the next step. Otherwise, return no match and release the read lock in the lock management area;

[0063] 4. Check the value recorded in the hash bucket in the data area to see if the in-memory data is consistent with the cached data being queried. If so, proceed to the next step.

[0064] Otherwise, continue searching, and if no matching data is found, return "unmatched" and release the read lock in the lock management area;

[0065] 5. Update the data access time in the algorithm management area;

[0066] 6. Return the matching result and release the read lock in the lock management area.

[0067] In order to support efficient query and update of hundreds of millions of data, the data storage format is either a red-black tree or a hash table. Considering the O(1) time complexity of the hash table under optimistic conditions, this application uses a hash table to store data. Since the data is stored in a hash table, the query and update time complexity is ideally O(1), so the query and update performance is very high.

[0068] Furthermore, in order to support concurrent query updates, locking control is required when operating data. If mutual exclusion locks are added to both data reading and writing, the lock range will be expanded. Reading data will not cause the data to change, so there is no need for mutual exclusion between reads. Just add a shared lock, which can greatly improve data query efficiency.

[0069] Furthermore, to ensure that data is not lost when the process exits (normally or abnormally), one approach is to store data in shared memory. However, if the host is restarted, the shared memory data will be lost.

[0070] Another way is to synchronize the shared memory data to the disk file. This application adopts the method of synchronizing the shared memory data to the disk file (with the help of the automatic write-back mechanism provided by the operating system).

[0071] However, a lot of memory fragmentation is caused by inconsistent allocation and release sizes. In order to avoid memory fragmentation, this application adds constraints. The size of the slots pre-allocated for storing data in shared memory is fixed, so the size of the memory space written and deleted each time is the same, and there will be no memory fragmentation. In view of the fact that the lengths of many CDN request URLs are inconsistent, when storing them in memory, the MD5 signature calculation is performed on the URL to obtain the MD5 value. Since the MD5 value is fixed at 32 bits, this ensures that the number of bits written to the memory is fixed. The MD5 signature calculation is also performed on the URL during query.

[0072] Furthermore, since the host's memory is fixed, the memory occupied by the hash table cannot be expanded indefinitely. The total size of the stored data needs to be controlled. When the written data exceeds the total size, this solution supports the data elimination algorithm LRU to eliminate historical data and then write new data.

[0073] Due to the complex CDN online environment, sometimes the CDN gateway process is terminated while writing to the hash table or exits abnormally due to other reasons, resulting in incomplete transactions. In addition, the abnormal transaction lock is not released, which also blocks other processes. This solution adds transaction management to roll back abnormally interrupted transactions.

[0074] To prevent the created hash table shared memory from being abnormally overwritten by other modules, such as wild pointers or buffer overflows, this solution write-protects the shared memory pages of the hash table. Only modules in this solution have permission to overwrite the hash table memory. Once other modules overwrite it, an exception error is thrown.

[0075] Among them, a wild pointer refers to an uninitialized pointer. The system will default to a pointer with a random address. If a wild pointer is used carelessly, it is easy to cause a segmentation fault due to memory leaks; a buffer overflow is an abnormal phenomenon, which means that when a program tries to put more data into a buffer, the data exceeds the capacity of the buffer itself, resulting in data corruption, program crashes, etc.; writing dirty shared memory cache data means that when multiple processes share the same block of memory, one process modifies the data in this memory without the other processes realizing it, resulting in data inconsistency.

[0076] The foregoing is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained herein shall, unless otherwise specified or limited, be implemented in accordance with conventional means in the art.

Claims

1. A method for supporting local data caching and high-concurrency secure access, characterized in that: include: Create a shared memory using the mmap function provided by the operating system, wherein the shared memory is mapped to a corresponding specific file on the disk file; When data is written to the shared memory, the operating system synchronizes the shared memory data to the disk file in real time, wherein the write cache data flow and query data flow between the shared memory and the disk file can be realized; When the cached data exceeds the shared memory capacity, the LRU elimination algorithm automatically deletes the old data and then writes the new data; Among them, the shared memory is divided into lock management area, data area, hash bucket area, algorithm management area, and transaction management area; Furthermore, the lock management area is used to control the modification of the critical section data of the hash table, ensuring that only one process can modify the critical section data at the same time. When reading and writing data, a shared lock is added through the lock management area. The transaction management area is used to ensure whether modifications are executed or not, avoid the generation of intermediate states, and is responsible for clearing unreleased locks and rolling back intermediate state transactions when the process exits abnormally. The data area is a fixed-length array, each element in the array is of equal size, and the elements store specific data written externally and data metadata information; The method of synchronizing the shared memory data to the disk file is achieved through the automatic write-back mechanism provided by the operating system; The size of the card slot pre-allocated for storing data in the shared memory is fixed; When the shared memory writes cache data flow and queries data flow, the uniform resource locator url is calculated to obtain the MD5 value; Memory fragmentation is caused by inconsistent allocation and release sizes. To avoid memory fragmentation, constraints are added to fix the size of the pre-allocated data storage slots in the shared memory. Therefore, the size of the memory space written and deleted each time is the same, and there will be no memory fragmentation. In view of the inconsistent length of the uniform resource locator URLs of many requests to the content delivery network CDN, when storing them in the memory, the uniform resource locator URL is calculated to obtain the MD5 value. Since the MD5 value is fixed at 32 bits, this ensures that the number of bits written to the memory is fixed. The MD5 signature calculation is also performed on the uniform resource locator URL during query.

2. The method for supporting local data caching and high-concurrency secure access according to claim 1, characterized in that: The hash bucket area is a row of buckets arranged continuously, and the value stored in each bucket is the data subscript of the data area above.

3. The method for supporting local data caching and high-concurrency secure access according to claim 1, characterized in that: The algorithm management area is used to record the order in which each data is accessed. When the memory is full of data, the earliest written but not accessed data is eliminated.

4. The method for supporting local data caching and high-concurrency secure access according to claim 1, characterized in that: The process of writing cache data includes: Step 1: Add a write lock in the lock management area, start a transaction in the transaction management area, and temporarily cancel the write protection for the hash bucket memory page; Step 2: Calculate the hash value of the cached data according to the hash function; Step 3: Check the corresponding hash bucket based on the hash value to see if the bucket has been written. If it has been written, go to the next step; otherwise, go to step 5. Step 4: Check the value recorded in the hash bucket to see if the memory data is consistent with the written cache data. If they are consistent, update the data directly and go to step 8. Otherwise, continue searching. If no matching data is found, go to the next step. Step 5. Find free space in the data area. If there is no free space, go to the next step. Otherwise, go to step 7. Step 6: Eliminate the oldest historical record from the algorithm management area and put it into the free space of the data area; Step 7: Request memory from the free space in the data area and write the cached data, and store the cached data in the hash bucket at the index position of the data area; Step 8: End the transaction in the transaction management area, release the write lock in the lock management area, and restore the write protection of the hash bucket memory page.

5. The method for supporting local data caching and high-concurrency secure access according to claim 1, characterized in that: The data query process includes: S1. Add a read lock to the lock management area; S2. Calculate the hash value of the cached data based on the hash function; S3. Check the corresponding hash bucket based on the hash value to see if there is a value in the bucket. If there is, proceed to the next step. Otherwise, return "no match" and release the read lock in the lock management area. S4. Check the value in the hash bucket to see if the in-memory data in the data area is consistent with the cached data. If so, proceed to the next step. Otherwise, continue searching, and if no matching data is found, return "unmatched" and release the read lock in the lock management area; S5. Update data access time in the algorithm management area; S6. Return the matching result and release the read lock in the lock management area.

6. The method for supporting local data caching and high-concurrency secure access according to claim 1, characterized in that: The lock management area writes a lock on the shared memory page of the hash table and performs write protection on the shared memory page.

Images