Storage engine control method, system and storage medium supporting large field data

Abstract

The application discloses a storage engine control method and system supporting large field data and a storage medium, constructs a memory pool for dynamically allocating and releasing memory blocks, the memory pool comprises a plurality of memory blocks stored in a linked list, and the memory block comprises a pointer pointing to a memory position; when large field data needs to be stored, a memory block with a corresponding size is allocated on demand. According to the large field data to be processed, a corresponding table structure and data type are designed; an index structure and a query strategy are determined; the query method comprises insert and find methods, for the large field data type, a B+ tree index structure is used for range query, and a hash index structure is used for point query; a data compression and cache strategy is determined, which is used for reducing memory occupation and improving query performance. The application successfully solves the problem that the memory storage engine in the prior art does not support large field data, improves the SQL penetration performance of the internal and external networks, and has good practicability.

Description

Technical Field

[0001] This invention belongs to the technical field of data exchange between internal and external networks, and specifically relates to a storage engine control method, system, and storage medium that supports large field data. Background Technology

[0002] In modern computer network environments, network tunneling is a common requirement for enabling data access and communication across network boundaries. SQL tunneling is a primary method for exchanging data between internal and external networks, allowing users to query and manipulate remote databases using SQL. The common database system MySQL supports multiple storage engines to meet different application scenarios and performance requirements. The Memory storage engine offers the following advantages:

[0003] (1) Data is stored in memory: The Memory storage engine stores all data in memory, without needing to write it to disk, thus resulting in high query performance.

[0004] (2) Fixed-size memory blocks: The Memory storage engine stores data in fixed-size memory blocks. This implementation simplifies memory management but limits support for large field data.

[0005] (3) Use hash index: The Memory storage engine uses hash index by default, which is suitable for equality queries, but does not support range queries.

[0006] Currently, MySQL's memory storage engine offers high performance because it stores data in memory for swapping. The Memory storage engine is a memory-based storage engine provided by MySQL where all data is stored in memory and does not need to be written to disk. This storage engine is characterized by high performance, but due to its internal implementation, it does not support large field data. This limits the application of the Memory storage engine in scenarios involving SQL penetration between internal and external networks. Existing technologies attempt to support large field data on top of the memory storage engine, for example:

[0007] (1) Dynamically allocate memory blocks: Try to support large field data by dynamically allocating memory blocks.

[0008] (2) Use B+ tree index: Replace the hash index of the Memory storage engine with a B+ tree index to support range queries.

[0009] In summary, the existing technology has the following disadvantages:

[0010] (1) Does not support large field data: Because the Memory storage engine uses fixed-size memory blocks to store data, it encounters limitations when processing large field data. This makes the Memory storage engine unsuitable for scenarios involving internal and external network SQL penetration that require processing large field data.

[0011] (2) Insufficient query performance: Existing technical solutions may not have adequately optimized the table structure, data types, and index structure, thus affecting query performance. For example, some solutions use B+ tree indexes to support range queries, but do not consider other optimization strategies, such as data compression and caching strategies.

[0012] (3) Memory management complexity: In existing technical solutions, some methods to support large field data are to dynamically allocate memory blocks, but this may increase the complexity of memory management and affect the overall performance.

[0013] Therefore, this invention aims to propose an improved memory storage engine to support large-field data and improve the performance of SQL penetration between internal and external networks. This invention employs an extended or self-developed storage engine, which, while inheriting the advantages of the memory storage engine's memory-swapping mechanism, adds support for large fields. In this way, when implementing internal and external network penetration, it fully utilizes the high-performance characteristics of the memory storage engine while solving its lack of support for large fields, thereby effectively improving the performance of internal and external network penetration. Summary of the Invention

[0014] The purpose of this invention is to provide a storage engine control method, system, and storage medium that support large field data, in order to solve the aforementioned problems.

[0015] This invention is mainly achieved through the following technical solutions:

[0016] A storage engine control method that supports large field data includes the following steps:

[0017] Step S1: Construct a memory pool for dynamically allocating and releasing memory blocks. The memory pool includes several memory blocks stored in a linked list. Each memory block includes a pointer to a memory location. When large field data needs to be stored, allocate memory blocks of the appropriate size as needed.

[0018] Step S2: Design the corresponding table structure and data types according to the large field data to be processed; define two new data types, LARGE_VARCHAR and LARGE_TEXT, corresponding to variable-length string data and text data respectively, to store large field data; define the data type and maximum size maxSize for each column of data. When inserting a row of data, if the data type is LARGE_VARCHAR or LARGE_TEXT, then insert the data in this column according to the processing method for large field data.

[0019] Step S3: Determine the index structure and query strategy; query methods include insert and find methods. For large field data types, use a B+ tree index structure for range queries and a hash index structure for point queries.

[0020] Step S4: Determine data compression and caching strategies to reduce memory usage and improve query performance.

[0021] To better implement the present invention, further, in step S1, when initializing the memory pool, a set size of memory is pre-allocated in the constructor, and a memory block is created with the set size as the memory block size and the pointer points to the allocated memory, and the memory block is added to the linked list;

[0022] When dynamically allocating memory blocks, the system receives the parameter `size` of the required memory block, then traverses the linked list to find a memory block whose size is greater than or equal to `size`. If a suitable memory block is found, memory of size `size` is allocated, and the size and pointer of this memory block are updated. If the size of the memory block is 0, the memory block is removed from the linked list. If no suitable memory block is found, an exception is issued.

[0023] When releasing a memory block, it receives a pointer ptr to the memory block to be released and the size size of the memory block, creates a new pointer to ptr and a memory block of size size, and then adds the memory block to the linked list; when the memory pool is no longer in use, it releases all memory blocks in the constructor.

[0024] To better implement the present invention, further, in step S2, a Large Field class is created for managing the segmented data blocks; in the Large Field class, a member variable, a vector of blocks, is defined for storing the segmented data blocks; in the Large Field class, when data is inserted, the original data is divided into multiple small blocks, and then these small blocks are copied to newly allocated memory blocks respectively, and the pointers of the newly allocated memory blocks are stored in the vector of blocks; when reading, data is copied from each memory block in sequence so that the original data can be reconstructed when needed.

[0025] To better implement the present invention, further, in step S3, in the insert method, after selecting the index structure and completing the insertion of the index entries, the index maintenance strategy is executed. By checking the current memory occupied by the index and the query performance, it is determined whether maintenance operations are needed. If the current memory occupied by the index exceeds the set threshold, the index entries are deleted to release the memory. If the query performance of the index is detected to be degraded, the index is rebuilt.

[0026] To better implement the present invention, further, in step S4, data compression and decompression are performed using the compress and decompress functions respectively:

[0027] In the `compress` function, first, the length of the original data, `src Len`, is obtained. Then, the `compress Bound` function of zlib is called to calculate the maximum possible size of the compressed data, `dest Len`, based on the size of the source data, and this value is used to create the buffer `dest`. Then, the `compress` function of zlib is used to compress the original data into the `dest` buffer. If the compression operation is successful, the `compress` function will return `Z_OK`, returning the compressed data; if the compression operation fails, an empty string is returned.

[0028] In the `decompress` function, firstly, it is assumed that the size of the decompressed data will not exceed 10 times the size of the original data, so a buffer `dest` 10 times the size of the original data is created; then, the `uncompress` function of zlib is used to decompress the compressed data into the `dest` buffer; if the decompression operation is successful, the `uncompress` function will return `Z_OK`, and the decompressed data will be returned; if the decompression operation fails, an empty string will be returned.

[0029] To better implement the present invention, further, in step S4, the least frequently used data items are deleted using the LRU strategy to ensure that the most frequently used data items are always retained in the cache; a Cache class is defined, and the Cache class includes a hash table cache and a doubly linked list data. The hash table cache is used to provide fast access for each key-value pair, and the doubly linked list data is used to represent the usage order of items in the cache, with the head of the list being the most recently accessed item and the tail being the least accessed item.

[0030] To better implement this invention, the `insert` function is used to insert new key-value pairs into the cache. If the key already exists in the cache, its corresponding value is updated, and the key-value pair is moved to the head of the linked list to indicate that it is the most recently used key-value pair. If the key does not exist in the cache, the new key-value pair is inserted at the head of the linked list, and an entry for the key-value pair is added to the hash table. If the cache capacity exceeds the preset maximum capacity after inserting the new key-value pair, the function will delete the least used key-value pair at the tail of the linked list and delete the corresponding entry in the hash table.

[0031] To better implement the present invention, further, in step S4, data is queried using the `find` public function. When querying data, it is first checked whether the required data item is already in the cache. If the key is found in the cache, a pointer to the corresponding value is returned, and this key-value pair is moved to the head of the linked list to indicate that it is the most recently used item. If the data item is not found in the cache, it is necessary to query from the data storage and decompress the query result. Then, the query result is inserted into the cache so that subsequent queries can retrieve data more quickly.

[0032] This invention is mainly achieved through the following technical solutions:

[0033] A storage engine system supporting large-field data, employing the aforementioned method, includes a data storage module, a memory management module, a query optimization module, and a data compression module interconnected via interfaces. The data storage module stores database data based on memory blocks; the memory management module dynamically allocates and releases memory blocks; the query optimization module selects the optimal execution strategy, optimal index structure, and query plan for a given query; and the data compression module compresses and decompresses data to reduce memory usage and improve query performance.

[0034] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described method.

[0035] The beneficial effects of this invention are as follows:

[0036] (1) This invention has successfully solved the problem that the memory storage engine does not support large field data in the prior art by improving memory management strategies, modifying table structure and data type support, and adjusting index and query optimization strategies, and has improved the performance of SQL penetration between internal and external networks. It has great practical value.

[0037] (2) This invention stores data in memory for exchange, thereby significantly improving the performance of internal and external network penetration. This invention proposes a strategy for dynamically allocating memory blocks to store large-field data. This invention adjusts the table structure and data types to support large-field data. This invention effectively solves the problem that the original memory storage engine does not support large fields, improving the performance of internal and external network SQL penetration;

[0038] (3) This invention employs appropriate index structures and query optimization strategies to adapt to the storage and query requirements of large-field data. These strategies improve query performance while also ensuring data security and consistency;

[0039] (4) This invention utilizes data compression technology and caching strategies. Data compression technology reduces memory usage, and caching strategies further improve query performance, thereby increasing memory utilization and the performance of SQL penetration between internal and external networks. These technical means can be adjusted according to actual needs to adapt to different scenarios;

[0040] (5) This invention is compatible with existing database systems (such as MySQL and Oracle), allowing users to seamlessly apply it to existing database systems, thereby improving the performance of SQL penetration between internal and external networks. The technical solution of this invention has excellent scalability and versatility, and can be applied to other similar in-memory storage engines to support large field data and improve performance. Attached Figure Description

[0041] Figure 1 This is a timing diagram for data exchange between internal and external networks. Detailed Implementation

[0042] Example 1:

[0043] This document outlines a storage engine control method to support large-field data and improve SQL penetration performance across internal and external networks. It includes the following steps:

[0044] (1) Improve memory management strategy: When storing large fields of data, memory blocks of appropriate size can be allocated on demand to avoid wasting memory resources. At the same time, a reasonable memory reclamation mechanism can effectively reduce memory fragmentation. To support large fields of data, we need to modify the memory management strategy of the Memory storage engine to allow dynamic allocation of memory blocks.

[0045] ① Dynamic memory management strategies rely on a memory pool. A memory pool is a pre-allocated block of memory from which memory blocks can be dynamically allocated and released. The design and implementation of the memory pool are crucial, as they directly impact the efficiency and performance of the memory management strategy. In this invention, the memory pool is designed to consist of a series of memory blocks. Each memory block contains a size and a pointer (ptr) to a memory location. These memory blocks are stored in a linked list, defined as a member variable of the MemoryPool class.

[0046] ② When initializing the memory pool, a certain size (totalSize) of memory is pre-allocated in the constructor. This operation is accomplished by calling the malloc function, which simultaneously creates a memory block of size totalSize, whose pointer points to the allocated memory, and adds this memory block to the linked list.

[0047] ③ The dynamic allocation of memory blocks is implemented in the `allocate` method. This method accepts a parameter `size`, representing the size of the required memory block. The `allocate` method traverses the linked list, searching for the first memory block whose size is greater than or equal to `size`. If such a memory block is found, it allocates memory of size `size` from it and then updates the size and pointer of this memory block. If the size of this memory block becomes 0, it is removed from the linked list. If no sufficiently large memory block is found, the `allocate` method throws a `bad_alloc` exception.

[0048] ④ The operation of freeing memory blocks is implemented in the `deallocate` method. This method accepts two parameters: a pointer `ptr` to the memory block to be freed, and the size of the memory block `size`. The `deallocate` method creates a new memory block of size `size`, whose pointer points to `ptr`, and then adds this memory block to the linked list. It's important to note that the `deallocate` method does not merge adjacent free blocks, which may lead to memory fragmentation.

[0049] ⑤ When the memory pool is no longer in use, all memory blocks are released in the constructor. This operation is accomplished by calling the free function.

[0050] ⑥ In the Enhanced Memory Storage Engine class, the memory pool described above is used for memory allocation and deallocation. When memory needs to be allocated, the memory pool's allocate method is called; when memory needs to be released, the memory pool's deallocate method is called.

[0051] (2) Modify table structure and data type support: Design appropriate table structures and data types according to the large field data to be processed. For example, large field data can be split into multiple smaller data blocks and stored in separate columns. Furthermore, appropriate data types can be defined for these columns to efficiently store and query large field data. To support large field data, we need to modify the table structure definition and related data manipulation functions.

[0052] 1) First, new data types were defined for large field data. In the EnhancedMemoryTable::DataType enumeration, two new data types, LARGE_VARCHAR and LARGE_TEXT, were added to store large field data. These two data types correspond to variable-length string data and text data, respectively, and are designed to handle data larger than ordinary strings or text.

[0053] 2) In the EnhancedMemoryTable::Column structure, a data type `type` and a maximum size `maxSize` are defined for each column. `type` is a DataType enumeration type, representing the data type of that column, which can include...

[0054] Any type, including LARGE_VARCHAR and LARGE_TEXT. maxSize is of type size_t and represents the maximum size of the data in this column. This is very important for handling large field data because it determines the maximum length of data that can be stored in this column.

[0055] 3) The insert method handles the insertion of large data fields. Specifically, when inserting a row, the data type of each column is used to determine how to process it. If the type is LARGE_VARCHAR or LARGE_TEXT, then the data in that column needs to be inserted according to the processing method for large data fields.

[0056] Preferably, for particularly large field data, it can be divided into multiple smaller blocks during writing and then written to different memory blocks respectively. During reading, these smaller blocks are then reassembled into complete data in sequence. The specific segmentation method can be determined according to actual needs; for example, it can be segmented according to a fixed size or according to the characteristics of the data. A class named LargeField is created to manage the segmented data blocks. In this class, a member variable `blocks` is defined, which is a pointer vector used to store the segmented data blocks. The LargeField class has an `insert` method for inserting data. When inserting data, the original data is divided into multiple smaller blocks, and then these smaller blocks are copied to newly allocated memory blocks, with pointers to the newly allocated memory blocks stored in the `blocks` vector. The size of the data block is assumed to be 4096 bytes in this example, but it can be adjusted according to actual needs. In the `retrieve` method, data is copied sequentially from each data block so that the original data can be reconstructed when needed.

[0057] (3) Adjust indexing and query optimization strategies: For large field data types, choose an appropriate index structure (such as a B+ tree or hash index) and optimize the query plan. For example, a B+ tree index can be used for range queries, while a hash index can be used for point queries. Adjust the indexing and query optimization strategies to adapt to large field data types.

[0058] The `EnhancedMemoryIndex` class is responsible for managing and using the index. A private index structure is defined within the class.

[0059] BPlusTree and HashIndex, as well as two main methods: insert and find.

[0060] B. In the `insert` method, insert index entries. The specific index structure used depends on the data type and use case. For example, if the data is a large field and range queries are required, a B+ tree index might be chosen. Conversely, if point queries are primarily performed, a hash index might be used. This decision can be based on performance testing or an understanding of the specific workload.

[0061] C. In the `find` method, a suitable index structure is selected based on the data type and query type. The specific selection principles are the same as in the `insert` method, also based on data type and query type. This method of optimizing the query plan ensures that the most suitable index is selected according to the actual needs and scenario, thereby improving query performance.

[0062] D. In addition, a private method called `maintain Index` has been added to the `Enhanced Memory Index` class to execute the index maintenance strategy. In the `insert` method, after selecting a suitable index structure and inserting the index entries, the index maintenance strategy is executed. This strategy checks the current memory usage of the index and query performance to determine whether maintenance operations are needed, such as deleting some index entries to free up memory or rebuilding the index to improve query performance.

[0063] E. The specific maintenance strategy depends on the current memory usage of the index and query performance. For example, if the memory usage of an index exceeds a certain threshold, it may be necessary to delete some index entries to free up memory. If a decline in index query performance is detected, the index may be rebuilt to improve performance.

[0064] The Enhanced Memory Index class defines two thresholds: memory Threshold and performanceThreshold. It also defines a check Memory Usage function, which checks the memory currently being used by the index.

[0065] A `checkPerformance` function is defined to check the query performance of the index.

[0066] Then, the index maintenance strategy is implemented in the `maintain Index` function. First, it checks if the current memory usage of the index exceeds the memory threshold. If it does, some index entries need to be deleted to free up memory. Next, it checks if the index's query performance is below the performance threshold. If it is, the index needs to be rebuilt to improve query performance.

[0067] (4) Data Compression and Caching Strategies: To reduce memory usage and improve query performance, data compression and caching strategies can be implemented. For example, a compression algorithm suitable for large field data can be used to reduce memory usage. Furthermore, some data can be cached according to query requirements to reduce query latency. For example, the Enhanced Memory StorageEngine class can be implemented to compress, decompress, and cache data.

[0068] a. First, there are two private functions in the Enhanced Memory Storage Engine class: compress and

[0069] decompress is used to perform data compression and decompression, respectively.

[0070] ① The `compress` function: This function takes a parameter `data` of type `std::string`, representing the original data to be compressed. First, it obtains the length of the original data, `srcLen`. Then, it calls the `compressBound` function of zlib, which calculates the maximum possible size of the compressed data, `destLen`, based on the size of the source data, and uses this value to create a sufficiently large buffer, `dest`. Next, it uses the `compress` function of zlib to compress the original data into the `dest` buffer. If the compression operation is successful, the `compress` function will return `Z_OK`, returning the compressed data; if the compression operation fails, it will return an empty string.

[0071] ② The `decompress` function: This function takes a parameter `data` of type `std::string`, representing the compressed data to be decompressed. First, assuming the decompressed data size will not exceed 10 times the original size, a buffer `dest` is created, 10 times the size of the original data. Next, the `uncompress` function from zlib is used to decompress the compressed data into the `dest` buffer. If the decompression operation is successful, the `uncompress` function will return `Z_OK`, returning the decompressed data; if the decompression operation fails, it returns an empty string.

[0072] b. Next, a private member "cache" is defined to implement the data caching strategy. This invention uses an LRU (Least Recently Used) strategy to remove the least frequently used data items, ensuring that the most frequently used data items are always retained in the cache. In this implementation, the LRUCache class consists of two key parts: a hash table...

[0073] The `(unordered_map)cache` is a hash table that provides fast access to each key-value pair, while the doubly linked list `data` represents the order in which items are used in the cache, with the most recently accessed item at the head and the least recently accessed item at the tail.

[0074] ① The `insert` function: This function inserts new key-value pairs into the cache. If the key already exists in the cache, the function updates its corresponding value and moves the key-value pair to the head of the linked list, indicating that it is the most recently used. If the key does not exist in the cache, the function inserts the new key-value pair at the head of the linked list and adds an entry for this key-value pair to the hash table. If the cache capacity exceeds the preset maximum capacity after inserting a new key-value pair, the function deletes the least recently used key-value pair from the tail of the linked list and removes the corresponding entry from the hash table.

[0075] ② In this caching strategy, a doubly linked list is used to ensure that elements are sorted according to their frequency and order of use, while a hash table ensures fast element retrieval. The combination of these two methods allows this LRU caching strategy to achieve high efficiency in insertion, search, and deletion operations.

[0076] c. Finally, there is a public function `find` for retrieving data. When retrieving data, it first checks if the required data item is already in the cache. If the key is found in the cache, the function returns a pointer to the corresponding value and moves this key-value pair to the head of the linked list, indicating that it is the most recently used item. If the data item is not found in the cache, it needs to be retrieved from the data store, and the query results may need to be decompressed. The query results can then be inserted into the cache so that subsequent queries can retrieve data more quickly. This query process may need to be designed based on the actual data storage and caching strategy.

[0077] First, this invention proposes an improved memory management strategy and modifies table structure and data type support, enabling memory storage engines that previously did not support large fields to store large data fields. This improvement enhances SQL penetration performance across internal and external networks while maintaining the high-speed access advantage of memory storage engines. This advantage directly solves the problem of existing memory storage engines not supporting large data fields, meeting the needs of data exchange between internal and external networks.

[0078] Secondly, by adjusting the index structure and query optimization strategies, this invention improves query performance while ensuring data security and consistency. This advantage solves the problem of low performance of in-memory storage engines when processing large field data in existing technologies, and helps to improve the efficiency of SQL penetration between internal and external networks.

[0079] Finally, this invention employs data compression and caching strategies to improve memory utilization and the performance of SQL penetration between internal and external networks. This strategy can be adjusted according to actual needs to adapt to different scenarios. This advantage allows this invention to maintain performance while also possessing a certain degree of flexibility and versatility.

[0080] In summary, this invention successfully solves the problem of existing memory storage engines not supporting large field data by improving memory management strategies, modifying table structures and data type support, and adjusting index and query optimization strategies. It also improves the performance of SQL penetration between internal and external networks and has significant practical value.

[0081] Example 2:

[0082] The storage engine system supports large-field data and includes modules for data storage, memory management, query optimization, and data compression. These modules are interconnected through appropriate interfaces to achieve the overall functionality.

[0083] Data Storage Module: This is the most fundamental module in the system, used to store and retrieve data from the database. All other modules depend on the storage and retrieval operations of this module. For large fields of data, the data storage module can use specific data structures (such as the mentioned block linked list) to store this data and perform operations through the insert, search, and delete functions of the "EnhancedMemoryTable" class.

[0084] Memory Management Module: This module is responsible for dynamically allocating and releasing memory. Its main task is to dynamically allocate memory for large data fields while releasing memory blocks that are no longer needed. This can be achieved through the "MemoryPool" class, including its functions for allocating and releasing memory blocks.

[0085] Query Optimization Module: This module is responsible for selecting the optimal execution strategy for a given query, including choosing the most suitable index structure and query plan. For example, a B+ tree index can be used for range queries, while a hash index can be used for point queries. This can be achieved through the "EnhancedMemoryIndex" class, including its functions for inserting index entries and retrieving data.

[0086] Data Compression Module: This module is responsible for compressing and decompressing data to reduce memory usage and improve query performance. This can be achieved through the compression and decompression functions of the "Enhanced Memory Storage Engine" class.

[0087] Preferably, these modules are interconnected through appropriate interfaces to achieve overall functionality. For example, the data storage module may provide an interface for the memory management module to allocate and release memory; the query optimization module may call the data storage module's interface to read data; and the data compression module may compress data before the data storage module writes it to memory and then decompress it when reading the data. In this way, the operations of each module can be seamlessly combined, thereby providing a high-efficiency memory storage engine optimized for large-field data.

[0088] Preferably, to fully leverage the advantages of this invention in internal and external network SQL penetration scenarios, the improved memory storage engine can be integrated with existing internal and external network penetration firewalls. For example, the memory storage engine can be used as a cache layer for internal and external network data exchange, thereby enabling fast data exchange in memory and improving internal and external network penetration performance.

[0089] Preferably, the present invention is also compatible with the existing database system MySQL. To this end, corresponding drivers and interfaces can be designed to enable the use of the present invention's in-memory storage engine in existing database systems. This allows users to seamlessly apply the present invention to existing database systems, thereby improving SQL penetration performance across internal and external networks.

[0090] The technical solution of this invention has excellent scalability and versatility. For example, memory management strategies, table structure design, indexing, and query optimization strategies can be adjusted according to actual needs to adapt to different scenarios. Furthermore, the technical solution of this invention can also be applied to other similar in-memory storage engines to support large-field data and improve performance.

[0091] Example 3:

[0092] An internal / external network data exchange system uses the aforementioned memory storage engine system as a cache layer for internal / external network data exchange, thereby enabling fast data exchange in memory and improving internal / external network penetration performance. It mainly includes the following components:

[0093] Client: The client initiates SQL query requests and communicates with the internal and external network penetration firewall system.

[0094] Internal / external network penetration firewall: The firewall is responsible for receiving and processing SQL query requests from clients and determining the query source based on the data in the in-memory storage engine. The firewall is also responsible for retrieving data from the backend database (if necessary) and loading the data into the in-memory storage engine.

[0095] Enhanced Memory Storage Engine: This is the core component of this invention, responsible for storing and managing large-field data in memory. It employs a dynamic memory block allocation strategy, supports large-field data types, and uses efficient index structures such as B+ trees.

[0096] Backend Database: The backend database (such as MySQL or Oracle) serves as the data source when large fields of data are not stored in the in-memory storage engine. The firewall retrieves data from the backend database and loads it into the in-memory storage engine.

[0097] like Figure 1 As shown, the interaction steps of the internal and external network data exchange system are as follows:

[0098] 1) The client sends an SQL query request to the firewall.

[0099] 2) The firewall determines the query source based on the data in the memory storage engine.

[0100] 3) If the data is already in the memory storage engine, the firewall directly retrieves the query results from the memory storage engine.

[0101] 4) If the data is not in the memory storage engine, the firewall retrieves the data from the backend database and loads it into the memory storage engine.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A storage engine control method supporting large field data, characterized in that, Includes the following steps: Step S1: Construct a memory pool for dynamically allocating and releasing memory blocks. The memory pool includes several memory blocks stored in a linked list. Each memory block includes a pointer to a memory location. When large field data needs to be stored, allocate memory blocks of the appropriate size as needed. Step S2: Design the corresponding table structure and data types according to the large field data to be processed; define two new data types, LARGE_VARCHAR and LARGE_TEXT, corresponding to variable-length string data and text data respectively, to store large field data; define the data type and maximum size maxSize for each column of data. When inserting a row of data, if the data type is LARGE_VARCHAR or LARGE_TEXT, then insert the data in this column according to the processing method for large field data. Step S3: Determine the index structure and query strategy; query methods include insert and find methods. For large field data types, use a B+ tree index structure for range queries and a hash index structure for point queries. Step S4: Determine data compression and caching strategies to reduce memory usage and improve query performance; In step S2, a LargeField class is created to manage the segmented data blocks. In the LargeField class, a member variable, a vector of blocks, is defined to store the segmented data blocks. When inserting data, the original data is divided into multiple small blocks, and these small blocks are copied to newly allocated memory blocks respectively. The pointers of the newly allocated memory blocks are stored in the vector of blocks. When reading, data is copied from each memory block in sequence so that the original data can be reconstructed when needed.

2. The storage engine control method for supporting large field data according to claim 1, characterized in that, In step S1, when initializing the memory pool, a set size of memory is pre-allocated in the constructor, and a memory block is created with the set size as the memory block size and the pointer points to the allocated memory. The memory block is then added to the linked list. When dynamically allocating memory blocks, the system receives the parameter `size` of the required memory block, then traverses the linked list to find a memory block whose size is greater than or equal to `size`. If a suitable memory block is found, memory of size `size` is allocated, and the size and pointer of this memory block are updated. If the size of the memory block is 0, the memory block is removed from the linked list. If no suitable memory block is found, an exception is issued. When releasing a memory block, it receives a pointer ptr to the memory block to be released and the size size of the memory block, creates a new pointer to ptr and a memory block of size size, and then adds the memory block to the linked list; when the memory pool is no longer in use, it releases all memory blocks in the constructor.

3. The storage engine control method for supporting large field data according to claim 1, characterized in that, In step S3, in the insert method, after selecting the index structure and completing the insertion of the index items, the index maintenance strategy is executed. By checking the memory currently occupied by the index and the query performance, it is determined whether maintenance operations are needed. If the memory currently occupied by the index exceeds the set threshold, delete the index entry to release the memory; If a decrease in query performance is detected for an index, rebuild the index.

4. The storage engine control method for supporting large field data according to claim 1, characterized in that, In step S4, data compression and decompression are performed using the compress and decompress functions, respectively. In the `compress` function, first, the length of the original data, `srcLen`, is obtained. Then, the `compressBound` function of zlib is called to calculate the maximum possible size of the compressed data, `destLen`, based on the size of the source data, and this value is used to create the buffer `dest`. Then, the `compress` function of zlib is used to compress the original data into the `dest` buffer. If the compression operation is successful, the `compress` function will return `Z_OK`, returning the compressed data; if the compression operation fails, an empty string is returned. In the decompress function, firstly, it is assumed that the size of the decompressed data will not exceed 10 times the size of the original data, so a buffer dest is created that is 10 times the size of the original data. Then, the zlib uncompress function is used to decompress the compressed data into the dest buffer. If the decompression operation is successful, the uncompress function will return Z_OK, and the decompressed data will be returned. If the decompression operation fails, an empty string is returned.

5. The storage engine control method for supporting large field data according to claim 1, characterized in that, In step S4, the LRU strategy is used to delete the least frequently used data items to ensure that the most frequently used data items are always retained in the cache; a Cache class is defined, which includes a hash table cache and a doubly linked list data. The hash table cache is used to provide fast access for each key-value pair, and the doubly linked list data is used to represent the usage order of items in the cache, with the head of the list being the most recently accessed item and the tail being the least accessed item.

6. The storage engine control method for supporting large field data according to claim 5, characterized in that, The `insert` function inserts a new key-value pair into the cache. If the key already exists in the cache, its corresponding value is updated, and the key-value pair is moved to the head of the linked list to indicate that it is the most recently used key-value pair. If the key does not exist in the cache, the new key-value pair is inserted at the head of the linked list, and an entry for the key-value pair is added to the hash table. If the cache capacity exceeds the preset maximum capacity after inserting a new key-value pair, the function will delete the least used key-value pair at the tail of the linked list and delete the corresponding entry from the hash table.

7. A storage engine control method for supporting large field data according to claim 5 or 6, characterized in that, In step S4, data is queried using the `find` public function. When querying data, the system first checks whether the required data item is already in the cache. If the key is found in the cache, a pointer to the corresponding value is returned, and this key-value pair is moved to the head of the linked list to indicate that it is the most recently used item. If the data item is not found in the cache, it needs to be queried from the data storage, and the query result needs to be decompressed. Then, the query result is inserted into the cache so that subsequent queries can retrieve data more quickly.

8. A storage engine system supporting large field data, comprising the method described in any one of claims 1-7, characterized in that, This includes data storage modules, memory management modules, query optimization modules, and data compression modules that are interconnected via interfaces; The data storage module is used to store database data based on memory blocks, and the memory management module is used to dynamically allocate and release memory blocks; The query optimization module is used to select the optimal execution strategy, the optimal index structure, and the query plan for a given query; the data compression module is used to compress and decompress data to reduce memory usage and improve query performance.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method described in any one of claims 1-7.

Images