Data processing method, memory manager, storage engine, storage medium and program product

Abstract

The data processing technical scheme provided by the embodiment of the invention is applied to a memory manager, the memory manager comprises a memory container with a plurality of storage blocks, the storage blocks are used for storing data items, one data item is composed of a hash value of one piece of data and coded data, and the coded data is obtained by coding the piece of data. Therefore, when the target data needs to be stored, the hash value of the target data and the coded data form a data item to be written into a corresponding storage block in the memory container, and by adopting the data storage scheme, compression and storage of any complex data type and structure into the memory can be supported; and storage does not involve an object head, so that the garbage recycling cost can be reduced.

Description

Technical Field

[0001] This application relates to the field of computing technology, and in particular to a data processing method, memory manager, storage engine, storage medium, and program product. Background Technology

[0002] The Java Virtual Machine (JVM) is a virtual computer on which Java programs run, primarily responsible for memory management. During memory management, the JVM uses an automatic memory management system to manage memory allocation and deallocation. When allocating memory, each object has an object header in memory to store its metadata. This object header occupies a significant amount of memory, leading to substantial memory waste, especially with small objects. Furthermore, if there are many small objects, the efficiency of garbage collection to release memory will be poor. These problems are particularly pronounced in primary key indexes of column-oriented memory engines developed using the Java programming language, where they are amplified and severely impact program performance. Summary of the Invention

[0003] In view of the problems mentioned in the background art above, this application is made to provide a data processing method, memory manager, storage engine, storage medium and program product that solves or at least partially solves the above problems.

[0004] A first aspect of this application provides a data processing method applied to a memory manager containing a memory container having multiple storage blocks. The method includes:

[0005] Perform a hash calculation on the target data to obtain a hash value;

[0006] The target data is encoded to obtain encoded data; the encoding method is variable-length encoding.

[0007] The hash value and the encoded data are combined to form a data item and written into the target storage block; wherein the target storage block is one of the multiple storage blocks contained in the memory container.

[0008] A second aspect of this application provides another data processing method applied to a memory manager containing a mapping container and a memory container. The method includes:

[0009] In response to a query request, determine the hash value of the requested data;

[0010] Based on the hash value, find a target slot from the underlying table area contained in the mapping container;

[0011] Execute the corresponding query processing based on the type of the target slot;

[0012] The mapping container further includes an extended linked list area, and both the base table area and the extended blocks in the extended linked list area have multiple slots for storing data. The target slot is one of the multiple slots in the base table area.

[0013] The target slot is of type empty slot, data slot, or extended pointer slot; the empty slot does not contain any data; the data slot contains a hash value and a data item identifier for a data item, the data item identifier indicating the data item corresponding to the data item stored in the memory container, the data item consisting of the hash value and encoded data of the data item, the encoded data being obtained by encoding the data item; the extended pointer slot points to an extended block in the extended linked list area.

[0014] A third aspect of this application provides a memory manager comprising:

[0015] The hash module is used to perform hash calculations on data to obtain hash values;

[0016] The encoding module is used to encode data to obtain encoded data;

[0017] A memory container having multiple storage blocks; each storage block is used to store data items, and a data item consists of a hash value and encoded data of a data item.

[0018] The mapping container comprises a base table area and an extended linked list area. Each extended block in both the base table area and the extended linked list area has multiple slots for storing data. The slot types include empty slots, data slots, and extended pointer slots. Empty slots do not contain data. Data slots store a hash value and a data item identifier for a given data item, with the data item identifier indicating the corresponding data item stored in the memory container. Extended pointer slots point to an extended block within the extended linked list area.

[0019] The memory manager implements the data processing methods provided in this application through the hash module, the encoding module, the memory container module, and the mapping container.

[0020] A fourth aspect of this application provides a storage engine that includes the memory manager provided in this application.

[0021] A fifth aspect of this application provides an electronic device. This electronic device has the storage engine provided in this application.

[0022] A sixth aspect of this application provides a database program product. This database program product has the storage engine provided in this application.

[0023] A seventh aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a computer, can implement the method described in any of the preceding claims.

[0024] An eighth aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0025] In the technical solution provided in this application embodiment, the memory manager includes a memory container with multiple storage blocks. Each storage block stores data items. A data item consists of a hash value and encoded data of a data item, obtained by encoding the data. Therefore, when target data needs to be stored, the hash value and encoded data of the target data are combined to form a data item and written into a corresponding storage block in the memory container. This data storage scheme enables compressed storage of any complex data type and structure into memory, without involving the object header, thus reducing garbage collection costs (JVM GC costs). Furthermore, the memory manager also includes a mapping container for a base table area and an extended linked list area. The extended blocks in both the base table area and the extended linked list area have multiple slots. Each slot stores the hash value and data item identifier corresponding to a data item. The data item identifier indicates the data item corresponding to that data item stored in the memory container. Therefore, after writing the data item corresponding to the target data into a corresponding storage block in the memory container, the hash value and data item identifier of the target data are also written into a corresponding slot in the mapping container. This application's solution uses a data item identifier (of integer (int) data type) in a mapping container to represent a primitive data (of complex data types and / or structures, or key-value data, etc.) in a memory container. This eliminates the limitation in Java where passing non-primitive data types (such as complex data structure types) requires constructing an object. Furthermore, when data needs to be queried, a target slot can be found from the basic table area contained in the mapping container based on the hash value of the data to be queried, and then the corresponding query processing can be performed according to the type of the target slot. The target slot can be an empty slot, a data slot, or an extended pointer slot; an empty slot does not contain data; a data slot contains the hash value and data item identifier of a data; an extended pointer slot points to an extended block in an extended linked list area. This application's solution, combined with a mapping container, achieves fast and efficient data querying. For example, if a hash value matching the hash value of the data to be queried is not found in the mapping container, no result can be returned directly without further querying the memory container. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the structure of a memory manager provided in an embodiment of this application;

[0028] Figure 2 This is a schematic diagram of the structure of a mapping container provided in an embodiment of this application;

[0029] Figure 3 , Figure 4a , Figure 4b and Figure 4c This application provides a timing diagram illustrating the principle of writing data to the memory manager in an embodiment of the application.

[0030] Figure 5 and Figure 6 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0031] Figure 7 and Figure 8 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. Detailed Implementation

[0032] Before introducing this application, the terminology used in the embodiments of this application will be explained first. It should be understood that this explanation is for the purpose of making the embodiments of this application clearer and does not necessarily constitute a limitation on the embodiments of this application.

[0033] Java is a high-level, object-oriented programming language. Java is executed through the Java Virtual Machine (JVM), which allows Java programs to run on different hardware and operating system platforms.

[0034] The JVM is a specification for computing devices; it's a virtual computer implemented by simulating various computer functions on a real computer. Java's memory management is primarily handled by the JVM. Specifically, the JVM manages memory resources through an automatic memory management system. This system mainly includes two mechanisms: memory allocation and garbage collection (GC).

[0035] Garbage collection (GC): Used to manage memory reclamation. Specifically, garbage collection (GC) is a mechanism provided by the garbage collector in the JVM to periodically reclaim the memory space occupied by any object references during idle time. Thus, the JVM uses a garbage collector to automatically manage memory reclamation. The garbage collector's task is to identify objects in the heap that are no longer used by the application and release the memory occupied by these objects. Garbage collection in Java typically consists of the following stages:

[0036] 1) Marking: The garbage collector examines all active objects. It starts with a set of objects called roots (such as local variables and input parameters in the currently executing method, active threads, static fields, etc.), and through these references, it can find and mark all reachable (i.e., still in use) objects.

[0037] 2) Sweeping / Collection: After the marking phase, the garbage collector removes unmarked objects and reclaims the memory they occupy. This may leave gaps in memory, i.e., fragmentation.

[0038] 3) Compaction: To avoid memory fragmentation, some garbage collectors compact the remaining objects, moving them to one end of the heap memory so that contiguous memory space can be allocated for new objects.

[0039] The JVM provides various garbage collectors, such as Serial GC, Parallel GC, Concurrent MarkSweep (CMS) GC, Garbage-First (G1) GC, ZGC, and Shenandoah GC. Each garbage collector has its own characteristics and is suitable for different scenarios and system performance requirements. Choosing and tuning the garbage collector is a crucial part of Java application performance optimization; proper selection and configuration can significantly impact application responsiveness and throughput. While the default garbage collector is sufficient for most applications, for high-performance systems, developers may need to tune the garbage collector's behavior based on the application's specific workload.

[0040] Full Garbage Collection (FGC) refers to the complete garbage collection process performed in the JVM, which can be understood as garbage collection of the entire heap memory. During Java program runtime, the JVM divides memory into different memory regions such as the heap and stack, each with a different purpose. For a detailed explanation of how the JVM divides memory into different regions and the function of each region, please refer to existing related content. Full garbage collection involves cleaning up the entire heap memory; for example, FGC is triggered when heap space usage reaches a certain threshold (e.g., 80%).

[0041] Heap memory: Used to store object instances, it is shared by all threads and created when the JVM starts. It is the largest memory area managed by the JVM and also the primary memory area managed by the garbage collector. For efficient GC, the heap memory space is generally divided into three memory areas: Young Generation (YG), Old Generation (OG), and PermGen (PG). The Young Generation, also known as the young generation, is used to store newly created objects. The Young Generation is typically divided into one Eden space and two Survivor spaces. The two Survivor spaces contain one from Survivor space (also called the from space) and one to Survivor space (also called the to space). In practice, newly created objects are first stored in the Eden space of the Young Generation, which is also the main memory area of ​​the Young Generation. Objects in the Eden space that survive one GC are moved to the from space, objects in the from space that survive one GC are moved to the to space, and objects in the to space that survive one GC are moved to the from space. Most objects (temporary objects) are garbage collected in the young generation, while some objects that survive several garbage collections (such as 15 garbage collections) in the young generation are moved to the old generation. In other words, the old generation (also known as the old generation) is used to store long-lived objects. The permanent generation refers to the permanent storage area in memory, mainly storing class and metadata information. Classes are stored in the permanent generation when they are loaded. Unlike other areas that store object instances, the permanent generation is not cleaned up by the garbage collector during the main program's execution, which causes it to fill up as the number of classes stored increases. In Java 8, the permanent generation was removed and replaced by an area called the "metaspace". The metaspace is similar in nature to the permanent generation, but the biggest difference is that the metaspace is not in the virtual machine (JVM) but uses native memory; therefore, the size of the metaspace is only limited by native memory.

[0042] Stop-the-World (STW) is a phase in the garbage collection (GC) process where the JVM pauses all application threads (making the entire application appear to be stopped) to ensure the garbage collector can safely and consistently perform necessary memory management operations. STW events are necessary for all garbage collectors, even those touted as "nearly non-stop." STW occurs primarily because the garbage collector requires the application to be in a stable state while performing certain tasks, such as determining which objects are live during the marking phase or moving objects during the compaction phase. If application threads continue running during these processes, they may modify object references, interfering with the garbage collector's operations and leading to incorrect results. The duration of a STW depends on many factors, including heap size, the number of live objects, the type and configuration of the garbage collector, and the specific garbage collection phase in progress. Generally, STW events should be as short as possible, as prolonged pauses can impact application performance and response time.

[0043] Arena (memory allocation): A specialized and efficient memory management technique used to quickly allocate and release memory within a specific context. This technique is typically used in performance-critical applications where a large number of small objects with similar lifecycles need to be allocated rapidly. Arena memory allocation improves performance by reducing the overhead of memory allocation, particularly by avoiding frequent system calls and heap memory management operations. An arena (also known as a region, memory pool, or memory area) is usually a large, pre-allocated contiguous block of memory. Within this block, objects can be allocated contiguously and quickly. Because the memory is contiguous, memory fragmentation is avoided, thus improving both space and time efficiency.

[0044] HashMap: HashMap is a widely used data structure in the Java Collections Framework. It is an implementation of the Map interface based on a hash table. HashMap provides a mapping from keys to values, where each key can map to at most one value. Specifically, HashMap uses a hash table to store these key-value mappings, which allows it to provide constant-time performance in most cases, making it very efficient for get and put operations. Therefore, HashMap can be understood as a collection for storing key-value pairs.

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0046] Furthermore, some processes described in the specification, claims, and accompanying drawings of this application include multiple operations that appear in a specific order. These operations may be performed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0047] As mentioned earlier, memory management in the Java programming language is handled by the JVM. The JVM manages memory through the Automatic Memory Management System (AMS) for memory allocation and garbage collection (GC). During memory allocation, Java objects are allocated on the heap, and each object in the heap has an object header that stores metadata such as hash code, lock status, and GC flags. The object header typically occupies 8 to 16 bytes, which is a significant amount of memory. For small objects, this can lead to substantial memory waste.

[0048] While some related technical solutions have proposed object header compression techniques to reduce the memory space occupied by object headers, thus partially alleviating the problem of excessive memory usage, they cannot solve the problem of a large number of small objects. Furthermore, if there are many small objects, the efficiency of garbage collection to release memory will be poor, for reasons including but not limited to the following:

[0049] 1) Small objects with long lifecycles will enter the old generation in memory, and objects in the old generation can only be reclaimed when a full garbage collection (FGC) is triggered again.

[0050] 2) When performing reachability analysis, it is done on a per-object basis. The more small objects there are, the worse the GC efficiency will be, and the longer it will occupy the processor (CPU) time.

[0051] 3) In scenarios with large heap memory, more small objects will be stored. When Full GC is triggered due to insufficient memory, FGC needs to scan more small objects, and the Stop-the-World (STW) event will take longer.

[0052] The problems described in 1) to 3) above are particularly prominent in the primary key indexes of columnar storage engines (also known as column-oriented storage engines). To support caching more primary key index data in memory, such as a 128GB columnar storage node, hundreds of millions of key-value (KV) pairs often need to be cached. When KV is not a basic data type, the aforementioned problems frequently cause the columnar storage node's CPU usage to remain high, resulting in stop-the-world (STW) pauses lasting several to tens of seconds during Full GC. Specifically, for example, when the columnar storage engine is developed using the Java programming language, the GC problems described in 1) to 3) above are amplified when the memory table in the storage engine is used in scenarios with a large number of long-lived small objects, severely impacting program performance.

[0053] While some existing solutions can handle fewer data objects in memory, they only support basic data types like long integers, and their hash table implementations use linear probing, which leads to focusing issues and performance degradation when dealing with large amounts of data.

[0054] In order to solve the memory management problems in the existing Java programming language, this application provides a data processing technical solution. The technical solution provided by this application can be applied to storage engines (such as columnar storage engines) developed in the Java programming language to manage large-scale memory-intensive small object data such as primary key indexes that need to be maintained; in particular, this application mainly focuses on implementing Arena memory management in storage scenarios with large heap memory, long lifecycles, and a large number of small objects, thereby reducing memory storage space and reducing the cost of JVM GC.

[0055] The data processing technical solution provided in this application is described in detail below.

[0056] In the data processing technical solutions provided in this application, the various method embodiments are implemented based on the memory manager provided in this application. The memory manager can be applied to a corresponding storage engine, which can be, but is not limited to, a columnar storage engine implemented in the Java programming language. Figure 1A schematic diagram of the memory manager structure provided in this application is shown. This memory manager implements memory management based on Arena technology under Java, and is therefore also called the Arena Memory Manager. The memory manager provided in this application is a service provider used to provide storage services for objects (mainly a large number of small objects) to reduce the number of objects at the JVM level. Here, the objects referred to are data, such as key-value data, ordinary single data, or combinations of multiple single data, etc., without specific limitations.

[0057] like Figure 1 As shown, the memory manager includes an encoder module 11, a decoder module 12, a hash module 13, a comparator module 14, an allocation pointer 15, and a memory container 16. The encoder module 11 is mainly used to perform encoding operations, the decoder module 12 is mainly used to perform decoding operations, the hash module 12 is mainly used to perform hash calculation operations, the comparator module 14 is mainly used to read data stored in the memory manager and compare it with externally transmitted data (such as comparing whether the numerical type, binary data, etc., are equal), and the allocation pointer 15 is used to obtain the location where data needs to be written to the memory container 16.

[0058] Therefore, in this application, the encoding operations are all performed by the encoding module 11, the decoding operations are all performed by the decoding module 12, the hash calculation operations are all performed by the hash module 13, and the comparison operations are all performed by the comparison module 14.

[0059] For a detailed explanation of the functions of these modules, please refer to the relevant content below, which will not be elaborated here.

[0060] The aforementioned memory container 16 is a large, pre-allocated contiguous block of memory, so it can be understood as a memory block. For example, a large contiguous block of memory can be pre-allocated as the memory container 16 at program startup or when needed. Specifically, in a Java context, the memory container 16 can be a Java dynamic array (ArrayList) object containing multiple storage blocks. These storage blocks within the memory container 16 are also called byte blocks (byte[]), and they are also Java objects. Both the memory container 16 and its storage blocks are Java objects managed by the JVM and subject to garbage collection (GC).

[0061] In short, the memory container 16 contains multiple storage blocks, such as storage blocks 161, 162, and 163 shown. The sizes of the multiple storage blocks can all be the same; for example, each storage block can be several MB to tens of MB in size, specifically, each storage block can be 16 MB. Of course, in some other embodiments, the sizes of the multiple storage blocks can be different, and this application does not limit this. Preferably, the multiple storage blocks are of the same size. Storage blocks are used to store data items (also called data objects or objects). Furthermore, the storage blocks in the memory container 16 are allocated on demand; once a storage block is full, a new storage block is allocated.

[0062] In practice, each data item in the storage block is processed before being stored to minimize memory usage. This processing includes hashing using hash module 13 to calculate the corresponding hash value and encoding using encoding module 11 to obtain the corresponding encoded data (serialized data). Therefore, each data item in the storage block can contain the hash value and encoded data of a corresponding piece of data.

[0063] This proposed solution, combining encoding and hashing, allows a single storage block to store a relatively large number of data items (e.g., tens of thousands of data items). Furthermore, it enables the JVM to manage the lifecycle of data in large memory blocks using storage blocks as the unit of management. This helps reduce the number of objects scanned during garbage collection (GC) and improves GC performance. Traditional Java memory management allocates memory separately for each object, with one memory storage unit per object. For example, storing a long integer (long) object requires a 12-byte object header plus 8 bytes for the long data, totaling 20 bytes. If there are n long objects, n 20-byte memory storage units are needed. Typically, this memory is allocated separately, resulting in n 20-byte memory units. During GC, the JVM scans and analyzes these memory units as separate units for references and performs garbage collection, which is very costly.

[0064] Furthermore, during the storage of data items, a data item identifier is assigned to each stored data item. This identifier can be a data item ID, used to uniquely identify the corresponding data item in the Areana memory manager; therefore, it is referred to as the Areana item id in this application. The data item identifier includes the block identifier of the storage block where the data is stored and the storage offset (in blockoffset) of the data item within that storage block. The storage offset can be understood as the write start offset corresponding to the data item when it is written to the storage block. Specifically, the data item identifier can be, but is not limited to, a 4-byte integer (int) basic type. One byte is an 8-bit binary number, thus the data item identifier can be a 32-bit binary number. The high eight bits can be used to represent the block identifier (such as block id) of the corresponding storage block, and the remaining low 24 bits can be used to represent the storage offset (i.e., the write start offset) of the corresponding data item within that storage block.

[0065] In this application, the block identifier of a storage block can be, for example, a block ID, which can refer to a block sequence number. The block sequence numbers of multiple storage blocks in the memory container 16 can be numbered using an auto-incrementing method.

[0066] Based on the data item identifier (Arena item id) encoding described above, when the data item identifier is a 32-bit binary number, the memory block container 16 can have a maximum of 256 storage blocks. This is because: since the high eight bits of the data item identifier represent the block identifier of the corresponding storage block, and 8 bits can only represent a range of 0 to 255, a total of 256 cases, 0 to 255 can be used to distinguish storage blocks, so 8 bits can distinguish 256 storage blocks. Furthermore, the maximum memory size managed by the memory container 16 = storage block size × 256. For example, when the data item identifier is a 32-bit binary number, the memory container 16 can have a maximum of 256 storage blocks. If each storage block is 16MB in size, then the memory container 16 can manage a maximum of 4GB (16MB × 256) of memory.

[0067] It should be noted that the above statement considers 0 as a valid identifier. In other instances, 0 can be used as a special null value instead of a valid identifier. For example, a data item identifier of 0 (i.e., both the block identifier of the corresponding storage block and the storage offset of a data item within that storage block are 0) is considered a special null value, while other values ​​can be used as valid identifiers for data items.

[0068] Based on the above, the data item identifier (Arena item id) can be stored using the basic integer type `int`, or it can be passed as an integer type `int`. Using basic data types effectively avoids the creation of new objects in Java. For example, when passing the data item identifier, there is no need to perform data type conversion, thus avoiding the creation of new objects due to data conversion. The passing of data item identifiers (such as the passing of data item identifiers involved in the sorting function of the memory manager, the return of data item identifiers by the memory manager to the corresponding caller, etc.) and storage will be described in other embodiments below.

[0069] Based on the foregoing, the memory manager can perform the following memory allocation process for the target data that needs to be stored in memory container 16:

[0070] (11) Calculate the hash value of the target data and encode the target data to serialize the target data.

[0071] The encoding is implemented based on the value storage method supported by memory container 16, including variable-length encoding. Specifically, the supported value storage methods include:

[0072] ① Integer data: Integer data is first zigzag encoded and then stored as a variable-length integer type (varint) (compressed storage). Therefore, the value storage representation of this storage method is: long value(varint(zigzag K)).

[0073] Zigzag encoding maps signed integers to unsigned integers, which are then stored as long integer values ​​using varint encoding. K represents the target data; specifically, K can represent the entire target data (if the target data is a single data item) or a portion of the target data (if the target data is key-value data, it can represent the value within the key-value data).

[0074] ② Other data types: For other data types such as binary byte data, the length of the bytes type is first composed of varint encoded bytes type data. The bytes type stores data in the form of a sequence of bytes (such as binary form).

[0075] Therefore, the storage representation of this value storage method can be shown as follows:

[0076] bytes value len(varint)bytes value data(bytes).

[0077] Here, bytes value len(varint) represents the length of the bytes type encoded in varint, and bytes valuedata(bytes) represents the bytes type data.

[0078] For example, if the target data is key-value data, the primary key Key in the target data can be hashed to obtain the corresponding hash value, and the value in the target data can be encoded using the methods ① or ② mentioned above to obtain encoded data (serialized data).

[0079] For example, if the target data is non-key value data, such as complex data composed of a single data or multiple single data, the target data can be hashed to obtain the corresponding hash value, and the target data can be encoded using the methods ① or ② mentioned above to obtain the corresponding encoded data.

[0080] (12) Obtain the write position of the target data by allocating pointer 15, and write the hash value and encoded data corresponding to the target data into a data item item in the corresponding storage block of memory container 16 according to the write position.

[0081] The write location can be determined based on the current pointing information of the allocation pointer 15. The current pointing information of the allocation pointer 15 may include, but is not limited to, the first storage block it currently points to and the write start offset of that first storage block. The first storage block is one of multiple storage blocks in the memory block container.

[0082] In practice, after determining the current pointing information of the allocation pointer 15, the remaining space of the first storage block currently pointed to by the allocation pointer 15 can be determined first. For example, the remaining space of the first storage block can be determined based on its size and the corresponding write start offset. Then, it can be determined whether the remaining space of the first storage block meets the writing requirements of the target data. If the remaining space of the first storage block does not meet the writing requirements of the target data, the current pointing information of the allocation pointer 15 can be updated so that the allocation pointer 15 points to a newly allocated second storage block and the write start offset is the beginning position of the second storage block. The write position determined based on the updated pointing information of the allocation pointer 15 includes the block identifier and the beginning position of the second storage block. Based on this write position, the target data will be written into the storage block with the beginning position of the second storage block as the write start offset. Conversely, if the remaining space of the first storage block meets the writing requirements of the target data, the writing position is determined directly based on the current pointing information of the allocation pointer 15. The writing position includes the block identifier of the first storage block and the writing start offset of the first storage block. Based on the writing position, the target data will be written to the first storage block with the writing start offset contained therein as the writing starting point.

[0083] Writing the target data to the corresponding storage block can mean combining the hash value and encoded data corresponding to the target data into a single data item and writing it to the corresponding storage block. Specifically, during the writing process, the hash value can be followed by the encoded data, meaning the encoded data is written as the value.

[0084] For example, see Figure 1Assuming that the allocation pointer 15 currently points to storage block 162 and the write start offset is position 'a' of storage block 162, the remaining space of storage block 162 can be determined based on its size (e.g., 16MB) and position 'a'. If the remaining space of storage block 162 is less than the size of the target data, it is determined that storage block 162 is insufficient to store the target data. In this case, a new storage block will be allocated, for example, storage block 163. Then, based on storage block 163, the allocation pointer 15 can be reassigned to update its pointing information, so that the allocation pointer 15 points to storage block 163 and the write start offset is the beginning of storage block 163. Based on the updated pointing information of the allocation pointer 15, the write position can be determined, including the block identifier of storage block 163 and the beginning position of storage block 163 (the write start offset). Based on this determined write position, the target data will be written to storage block 163 starting from the beginning position of storage block 163. Conversely, if the remaining space of storage block 162 is equal to or greater than the size of the target data, then storage block 162 is deemed sufficient to store the target data. In this case, the write position, including the block identifier of storage block 162 and the position a (the write start offset) of storage block 162, can be directly determined based on the current pointer information of allocation pointer 15. Based on this determined write position, the target data will be written to storage block 162 starting from position a. When writing the target data to the corresponding storage block 163 or storage block 162, the hash value and encoded data (Value) of the target data are written, with the encoded data following the hash value.

[0085] In this application, for target data that needs to be written (inserted) into the memory container, the hash value of the target data is calculated and stored together with the encoded data of the target data in the corresponding storage block of the memory container during the write operation. The benefits of this include accelerating hash calculations. The specific reasons for accelerating hash calculations are as follows:

[0086] Generally, the hash calculation results for the same data are consistent. Storing the hash value as data in the storage block allows for prioritizing hash value equality during data comparison (e.g., ...). Figure 4b The comparison involved in the comparison process (if they are not equal, then the corresponding data is definitely not equal either). This saves the trouble of comparing the stored data item byte by byte with the query value to be compared, which can improve the speed of comparison or sorting by hash value. In the process of comparison or sorting, it is often necessary to obtain the hash value in the data item multiple times. However, this application directly stores the hash value in the storage block, so that the hash does not need to be calculated repeatedly in the process of comparison or sorting, thus achieving the function of accelerating hash calculation.

[0087] (13) After the data item corresponding to the target data is written, update the information pointed to by the allocation pointer 15. Specifically, the write start offset pointed to by the allocation pointer 15 can be updated to the write end position of the data item corresponding to the target data, while the target storage block it points to remains unchanged.

[0088] Continuing with the example given in (12), after writing the data item corresponding to the target data to storage block 163, the allocation pointer 15 can be reassigned so that it still points to storage block 163, but the write start offset is updated from the beginning of storage block 163 to the end of the write operation of the data item corresponding to the target data in storage block 163. Here, the end of the write operation can be determined according to the size of the data item corresponding to the target data. Alternatively, after writing the data item corresponding to the target data to storage block 162, the allocation pointer 15 can be reassigned so that it still points to storage block 162, but the write start offset is updated from position a of storage block 162 to the end of the write operation of the data item corresponding to the target data in storage block 162. Here, the end of the write operation can be determined according to position a and the size of the data item corresponding to the target data.

[0089] (14) Encode the block identifier (such as the block number) and the write start offset of the storage block to be written into an integer (int) value, and return it to the corresponding calling module as the data item identifier (Arena item id) of the target data.

[0090] Continuing with the example given in (12), if the data item corresponding to the target data is written to storage block 163, the block identifier of storage block 163 and the beginning position of storage block 163 can be encoded into an int value, which is returned to the corresponding calling module as the data item ID (Arena item id) of the target data. If the data item corresponding to the target data is written to storage block 162, the block identifier of storage block 162 and position 'a' can be encoded into an int value, which is returned to the corresponding calling module as the data item ID (Arena item id) of the target data.

[0091] The aforementioned calling module refers to the external service logic module. For example, after the external service logic module sends a write request to the Arena memory manager to store the target data into the corresponding storage block in memory container 16 of the Arena memory manager, the external service logic module later needs the data item ID (Arena item id) returned by the memory manager to retrieve the target data for other processing, such as comparing sizes or returning it to the front-end client.

[0092] Based on the above, this application is approved. Figure 5This diagram illustrates a data processing method executed by the aforementioned memory manager. The memory manager comprises a memory container with multiple storage blocks. For example... Figure 5 As shown, a data processing method executed by this memory manager may include the following steps:

[0093] 101. Perform a hash calculation on the target data to obtain the hash value;

[0094] 102. Encode the target data to obtain encoded data; the encoding method is variable-length encoding;

[0095] 103. The hash value corresponding to the target data and the encoded data are combined to form a data item and written into the target storage block; wherein the target storage block is one of the multiple storage blocks contained in the memory container.

[0096] For a detailed description of the implementation of 101-102, please refer to other embodiments. Figure 1 The relevant content described. For example, see the relevant content in (11) of the aforementioned “Memory Allocation Process”.

[0097] In a specific implementable solution, the above-mentioned 103 "combining the hash value corresponding to the target data and the encoded data into a data item and writing it into the target storage block" includes:

[0098] 1031. Based on the current pointer information in the memory manager, determine the target storage block and the write start offset of the target storage block;

[0099] 1032. Using the write start offset as the write starting point, write the hash value and the encoded data together to form a data item and write it to the target storage block;

[0100] 1033. After the data item is written, update the pointer information so that the write start offset pointed to by the pointer is updated to the write end position of the data item.

[0101] In specific implementation, the aforementioned pointing information includes the first storage block currently pointed to by the allocation pointer and the write start offset corresponding to the first storage block; and step 1041 may include the following steps:

[0102] 10311. Determine whether the remaining space of the first storage block meets the writing requirements of the target data;

[0103] 10312. If satisfied, the first storage block is determined as the target storage block, and the write start offset of the first storage block is determined as the write start offset of the target storage block.

[0104] 10313. If not satisfied, a free second storage block is allocated through the allocation pointer and determined as the target storage block, and the beginning position of the second storage block is determined as the write start offset of the target storage block;

[0105] The first storage block and the second storage block are storage blocks among the plurality of storage blocks.

[0106] For a detailed description of the implementation of steps 10311 to 10313 and steps 1032 to 1033, please refer to other embodiments. Figure 1 The relevant content described.

[0107] Furthermore, the memory manager also includes a mapping container for storing hash values ​​and corresponding data item identifiers to better support data storage deduplication and retrieval. This mapping container can be a HashSet or Map class container. HashSet and Map are two different container classes. Specifically, HashSet is an unordered container that uses a hash function to store keys for fast lookup and retrieval; Map also uses a hash function to store keys, but it stores key-value pairs, i.e., one key corresponds to one value. In this application, the aforementioned mapping container is also a Java object, implemented as a long integer array, consisting of only one Java object (and some auxiliary metadata variables, such as slot numbers as described in other embodiments below). The long array is a special type of array; specifically, it is a collection of several long type variables used to store long integer data (64 bits).

[0108] Therefore, in this application, the mapping container can be a hash map implemented using a long array. In specific implementation, to address the data aggregation problem inherent in using linear probing to implement a hash map, the mapping container 17 here is implemented using a chained list method. For a detailed explanation of the concepts of linear probing and chained lists, please refer to the existing content.

[0109] Furthermore, after writing the target data into the memory container, the hash value of the target data and the data item identifier (Arena item id) can also be written to the mapping container in pairs. Specific implementation details will be discussed in other embodiments below. Figure 2 The description provides details, so it will not be elaborated here.

[0110] Figure 2 A schematic diagram of the mapping container is shown. For example... Figure 2 As shown, the mapping container 17 is mainly divided into the following two areas: the basic table area 171 and the extended linked list area 172.

[0111] The aforementioned basic table area 171 can be understood as a basic hash table entry part, which is a dense hash table. Therefore, the multiple slots in basic table area 171 are called hash slots. A hash slot represents a single position in the hash table; that is, a position in the hash table is generally called a slot. Each slot can store a data element and is named with an integer (e.g., numbered starting from 0). Basic table area 171 consists of a set of long integers, with each long corresponding to one slot (i.e., one long corresponds to one hash slot). Thus, each slot in basic table area 171 is a long, and one long is 8 bytes, so each slot can have 64 bits. The high 32 bits and low 32 bits can be used to store different types of data.

[0112] Based on the data storage status of the slots, the slots in the basic table area 171 may have the following three types: data entry, empty entry, and index entry. A data entry is a slot that already stores a corresponding data pair. The stored data pair consists of the hash value of a data item and its corresponding data item identifier (such as data item ID). The hash value can be stored using the high 32 bits, and the data item identifier can be stored using the low 32 bits. An empty entry is a slot that does not store any data; its high and low 32 bits are both 0. An index entry is a slot that stores the offset of an extended block in the extended linked list area. For example, the low 32 bits can be used to store the corresponding extended block offset, while the high 32 bits are 0. Therefore, an index entry points to an extended block in the extended linked list area.

[0113] It should be noted that initially, all slots in the base table area 171 are empty. Also, the number of slots in the base table area 171 is fixed. This allows for unique location of a slot during subsequent write (insert) or query operations by taking the modulo of the hash result with the number of slots, enabling the corresponding operation to be performed based on the located slot. For details on the specific implementation of write operations based on the located slot, please refer to the data insertion content described in other embodiments. Similarly, for details on the specific implementation of query operations based on the located slot, please refer to the data retrieval content described in other embodiments.

[0114] The aforementioned extended linked list area 172 is used to store the linked list in case of hash collisions. For example, if multiple inserted data fall into the same slot in the base table area, a hash collision can be considered to have occurred. Each unit in the linked list consists of several long strings, thus each unit in the linked list can have several positions for storing data. When all positions are full, a new set of long strings is allocated to form a new unit and attached to the end of the linked list.

[0115] In this application, each unit in the aforementioned linked list is called an extension block, and the position within the unit is called a slot. Therefore, each extension block in the extended linked list area is also composed of multiple longs. Similarly, an extension block also has multiple slots, and the number of slots is fixed. For a detailed description of the slots in the extension block, please refer to the aforementioned description of slots in the base table area 171. If an extension block is insufficient for storage, its last slot points to the newly allocated next extension block. In this application, the size of the extension block can be equal to the cache line size. For example, the cache line size of a typical mainstream processor (CPU) is 64 bytes (i.e., 16 32-bit integers). 64 bytes is the smallest data unit that the CPU loads data from memory. Therefore, the specific size of the extension block can be 64 bytes. In this case, the extension block can have a total of 8 64-bit slots.

[0116] The following is combined with Figure 2 , Figure 3 up Figures 4a to 4c This section describes the data writing (data insertion) process performed by the memory manager. The details are as follows:

[0117] (21) After the memory manager receives a write request, as follows: Figure 3 As shown, the target data to be written will be hashed to obtain the hash value Hc of the target data.

[0118] During hash calculation, special processing can be used to ensure that a value of 0 is not generated, facilitating subsequent slot lookup and differentiation. This special processing could include, for example, specifying a hash value Hc of 1 if the hash calculation of the target data results in 0, thus guaranteeing that the calculated hash value will never be 0.

[0119] (22) According to the slot allocation rules, find a corresponding target slot in the basic table area 171 for the target data, and process the target data according to the target slot.

[0120] The above slot allocation rules may include: taking the modulo of the number of slots using the hash value (i.e., mod operation). Therefore, the specific implementation of (22) above may include the following steps:

[0121] a1. First, the hash value Hc of the target data can be moduloed by the number of slots in the base table area 171 (i.e., the hash value Hc of the target data mod the number of slots in the base table area 171) to obtain the corresponding remainder.

[0122] a2. Then, a target slot is determined based on the remainder. Specifically, the slot in the base table area 171 with the same number as the remainder is the target slot.

[0123] a3. Finally, determine the type of the target slot based on its data storage status.

[0124] For example, if the target slot contains no data, meaning all bits in the target slot are 0, then the target slot is an empty entry. If the target slot contains data, then the target slot may be a data entry or an index entry. In this case, the type can be further determined based on the status of the first few bits of the target slot. For example, based on the description of slots in other embodiments, when the slot is 64-bit, the high 32 bits and low 32 bits of the data slot both store data (i.e., both the high 32 bits and low 32 bits are non-zero), while the high 32 bits of the extended pointer slot are 0 and the low 32 bits store data (i.e., the low 32 bits are non-zero). Therefore, if there is data in the target slot, it can be further determined whether the high 32 bits of the target slot are all 0 (whether there is data). If they are all 0, the target slot is determined to be a data slot (data entry); otherwise, if they are not all 0, the target slot is determined to be an extended pointer slot (index entry).

[0125] Of course, if the target slot contains data, other methods can be used to determine whether the target slot is a data slot or an extended pointer slot. For example, the specific data content in the target slot can be read. If the specific data content contains a hash value and a data item identifier (data item ID), then the target slot is determined to be a data slot; if the specific data content read only contains an extended block offset, then the target slot is determined to be an extended pointer slot.

[0126] (23) Process the target data according to the type of the target slot. Specifically:

[0127] Scenario 1: The target slot is an empty slot.

[0128] See also Figure 4a As shown, if the target slot is empty, the following write operation will be performed on the target data:

[0129] The target data is written to the memory manager; specifically, it is written to the corresponding storage block within the memory container 16 contained in the memory manager. Furthermore, the target data can be encoded during the writing process, so that the hash value Hc of the target data and the encoded data are combined into a single data item and written to the corresponding storage block.

[0130] For a detailed implementation of the write operation described above for the target data, please refer to other embodiments. Figure 1 The relevant content described.

[0131] In addition, the hash value Hc of the target data and the data item identifier (data item ID) will be directly filled into the target slot. The data item identifier is obtained by encoding the block identifier of the storage block to which the target data is written and the write start offset into an int value. For details on obtaining the data item identifier, please refer to relevant content in other embodiments. After writing is complete, the target slot will change from an empty slot to a data slot.

[0132] Scenario 2: The target slot is a data slot

[0133] See also Figure 4b As shown, if the target slot is a data slot, the hash value Hc calculated for the target data can be compared with the hash value Hs stored in the target slot to obtain the first comparison result. Then, different processing procedures are performed based on the first comparison result. Specifically:

[0134] B1, The first comparison result is the same.

[0135] If the first comparison result shows that the calculated hash value Hc is the same as the hash value Hs stored in the target slot, then data is read from memory container 16 according to the data item identifier stored in the target slot to retrieve the corresponding data item. Specifically, the decoder 12 can decode the data item identifier stored in the target slot to read the corresponding data item from the corresponding storage block in memory container 16 according to the decoded block identifier and write start offset. Further, the decoder 12 can also decode the encoded data in the read data item to obtain the corresponding decoded data. The comparison module 14 then compares the decoded data and the target data to obtain a second comparison result. Different processing procedures are then performed based on the second comparison result. Specifically, as follows:

[0136] b11) If the second comparison result is that the decoded data is the same as the target data, it means that duplicate data writing has occurred, and the data duplication processing operation can be performed at this time.

[0137] The data duplication processing operation includes: outputting error messages or data replacement processing according to preset rules.

[0138] For example, combining Figure 1 Assuming the decoded data is obtained by decoding the encoded data contained in data item 622 in storage block 162, if the decoded data is the same as the target data, an error message can be output or data replacement can be performed on data item 622. The data replacement process can be as follows: output a prompt message to indicate whether duplicate data exists and whether to replace it; if the user triggers the replacement operation, the hash value Hc of the target data and the corresponding encoded data can be combined to form a new data item 622', and the original old data item 622 can be replaced with this new data item 622' using a method such as overwrite. The prompt message can be a text prompt or a voice prompt, etc., and is not specifically limited here.

[0139] (b12) If the second comparison result shows that the decoded data is different from the target data, a new extended block can be allocated in the extended linked list area 172, and the data content originally stored in the target slot can be migrated to the extended block for storage. The target slot can also be converted into an extended pointer slot to point to the extended block. Furthermore, the hash value Hc of the target data and its corresponding encoded data will be combined into a data item and written into the corresponding storage block in the memory container 16. The hash value of the target data and its corresponding data item identifier will also be written into an empty slot in the extended block.

[0140] For example, combining Figure 2 and Figure 1Assuming the target slot is a data slot, such as slot 712, in the base table area 171, and further assuming the decoded data is obtained by decoding the encoded data contained in data item 622 in storage block 162, if the decoded data is different from the target data, a new extended block 722 can be allocated in the extended linked list area 172, and at this time, the multiple slots in the new extended block 722 are all empty slots. Then, the data content in slot 712 can be migrated to the first slot in the extended block 722 for storage, which changes slot 712 from a data slot to an empty slot. Then, the offset of the extended block 722 in the extended linked list area 172 is filled into the lower 32 bits of slot 712, making slot 712 change from an empty slot to an extended pointer slot (which points to the extended block 722). After writing the hash value Hc of the target data and its corresponding encoded data into the corresponding storage block in memory container 16, the hash value Hc of the target data and its corresponding data item identifier are also written into an empty slot in extension block 722, for example, into the slot after the first slot in extension block 722. At this point, extension block 722 now contains two data records.

[0141] B2. The first comparison result is different.

[0142] If the first comparison result shows that the hash value Hc of the target data is different from the hash value Hs stored in the target slot, it means that the data content pointed to by the data item identifier stored in the target slot is different. The target data is new data that needs to be written. At this time, the following operation will be performed:

[0143] Allocate a new extended block in extended linked list area 172, migrate the data originally stored in the target slot to this extended block, and convert the target slot into an extended pointer slot pointing to this extended block. Additionally, perform the following operations on the target data: Figure 4a The given write operation process is to write the hash value Hc of the target data and the corresponding encoded data into a data item in the corresponding storage block of the memory container 16, and to write the hash value of the target data and the corresponding data item identifier into an empty slot in the extension block.

[0144] For details on the specific implementation of this content, please refer to the relevant content given in b12 above, which will not be elaborated here.

[0145] Scenario 3: The target slot is an extended pointing slot.

[0146] like Figure 4c If the target slot is an extended pointer slot in base table area 171, the following jump traversal operation will be executed:

[0147] C1. Based on the offset of the extension block stored in the target slot (the slot to which the extension is pointed), jump to the corresponding extension block in the extension list;

[0148] C2. Iterate through each slot in the expansion block. During the iteration, if a data slot is encountered, the hash value Hc of the data is compared with the hash value stored in the data slot.

[0149] C3. Perform the corresponding processing based on the traversal results. Specifically, as follows:

[0150] If, through traversal, it is determined that the hash value stored in a data entry within the extended block is the same as the hash value Hc of the target data, then the aforementioned combination... Figure 4b The content described in "B1, the first comparison result is the same" will be processed.

[0151] If, after traversal, it is determined that the hash values ​​stored in all data slots of the extended block are different from the hash value Hc of the target data, and an empty slot is encountered during the traversal, then the write operation content given in "Case 1" above can be used to write the hash value Hc of the target data and the corresponding encoded data into a data item and write it into the corresponding storage block. The hash value Hc of the target data and the data item identifier (data item ID) are directly filled into the empty slot encountered in the extended block. Here, the empty slot mentioned here is not limited to the first empty slot encountered during traversal.

[0152] If, after traversal, it is determined that the hash values ​​stored in all data slots in the extended block are different from the hash value Hc of the target data, and an extended pointing slot is encountered during the traversal (but no empty slot is encountered), then the aforementioned steps C1 to C3 are repeated.

[0153] This application's solution also stores the hash value in the mapping container (a hash table). This allows for the retrieval of the hash value without retrieving the original data from the memory container during rehashing of the mapping container, thus accelerating the rehashing process. Furthermore, it improves query speed and efficiency. For example, if a hash value matching the requested data is not found in the mapping container, no result can be found; there's no need to search the memory container again. The specific implementation of the rehashing and data querying described here is detailed in other embodiments below and will not be elaborated upon here.

[0154] In this application, when all slots in the base table area 171 or the extended block are non-empty slots (data slots or extended pointer slots), a new extended block is allocated in the extended linked list area. Following the relevant content given in b12 above, the data content in the last data slot (data entry) of the base table area 171 is migrated to the newly allocated extended block for storage. The offset of the newly allocated extended block in the extended linked list area is written to the original last data slot, updating it to an extended pointer slot pointing to the newly allocated extended block. Furthermore, the write operation process is also performed according to the methods given in other embodiments (such as...). Figure 4a As shown, the new data to be written is written to the corresponding storage block in memory container 16, and the hash value and data item identifier corresponding to the new data are written to the newly allocated extension block. Additionally, if there is no space to allocate a new extension block in the extended linked list area of ​​the mapping container, a rehash operation will be triggered. The rehash operation process is as follows:

[0155] 31) Construct a new long array as the new mapping container. This new mapping container can be larger than the original mapping container. For ease of description, the new mapping container will be referred to as the second mapping container, and the original mapping container will be referred to as the first mapping container.

[0156] 32) Traverse the old first mapping container and migrate the data content of all data slots in it to the new second mapping container. The migration is performed according to other embodiments. Figure 2 , Figure 3 up Figures 4a to 4c The described data writing (data insertion) process is implemented, and the hash value can be directly retrieved from the data slot and the data item identifier can be directly reused. Extended rehash can be completed without accessing the original data stored in the memory container contained in the memory manager.

[0157] For example, a data slot `data1` in the first mapping container contains a hash value `Hash1` and a data item identifier `ID1`. The hash value `Hash1` in this data slot can be directly used to calculate the remainder by taking the modulo of the number of slots in the base table area of ​​the second mapping container. Based on this remainder, a slot in the base table area of ​​the second mapping container can be found as the target slot. The data content in the data slot `data1` is then migrated according to the type of the target slot. For example, if the target slot is empty, the hash value `Hash1` and the data item identifier `ID1` from the data slot `data1` in the first mapping container are directly filled into the target slot found in the second mapping container, without needing to access the original data item identified by the data item identifier `ID1` stored in the memory container. For cases where the target slot is a data slot or an extended pointer slot, please refer to other embodiments for details. Figure 4b and Figure 4c The details of the description will not be elaborated here.

[0158] There is an above combination Figure 2 , Figure 3 as well as Figures 4a to 4c As described, the memory manager also includes a mapping container with a base table area and an extended linked list area. Both the base table area and the extended blocks in the extended linked list area have multiple slots for storing data. Furthermore, the data processing method performed by the memory manager may also include the following steps:

[0159] 104. Based on the hash value of the target data, find a target slot in the base table area;

[0160] 105. Based on the type of the target slot, perform corresponding processing operations on the target data.

[0161] The target slot is of type empty slot, data slot, or extended pointer slot; the empty slot does not contain any data; the data slot contains a hash value of a data and a data item identifier, the data item identifier indicating the data item corresponding to the data stored in the memory container; the extended pointer slot points to an extended block in the extended linked list area.

[0162] In step 104 above, the target slot can be determined by taking the remainder of the hash value of the target data modulo the number of slots in the base table area. Therefore, step 104 can be implemented as follows:

[0163] 1041. Use the hash value of the target data to calculate the remainder of the number of slots in the base table area;

[0164] 1042. Based on the remainder, find a target slot from the multiple slots contained in the basic table area. The target slot's number is the same as the remainder.

[0165] In the above 105, the processing operation includes, but is not limited to, at least one of the following: write operation, data repetition processing operation, jump traversal operation, etc.

[0166] The write operation includes: writing a data item into the target storage block by combining the hash value and the encoded data corresponding to the target data, and determining the data item identifier corresponding to the target data according to the block identifier of the target storage block and the write start offset, and filling the hash value and the data item identifier into a corresponding slot in the mapping container;

[0167] The data duplication processing operation includes outputting error messages or data replacement processing; the data replacement processing includes replacing a data item in the memory container indicated by the data item identifier stored in the data slot with a data item composed of the hash value and encoded data of the target data;

[0168] The jump traversal operation includes: jumping to the extension block pointed to by the extension pointer slot, and traversing each slot in the extension block in turn to perform corresponding processing according to the traversal result.

[0169] Furthermore, in a specific instance, if the target slot is an empty slot, then step 105 above, "based on the type of the target slot, perform corresponding processing operations on the target data," may include:

[0170] 1051a. Perform the write operation;

[0171] Specifically, when performing the write operation, the hash value of the target data and the corresponding data item identifier are filled into the target slot.

[0172] For a detailed description of the implementation of the above-mentioned 1051a, please refer to other embodiments. Figure 4a The relevant content described.

[0173] In another specific example, if the target slot is a data slot, then step 105 above, "based on the type of the target slot, perform corresponding processing operations on the target data," may include:

[0174] 1051b. Compare the hash value of the target data with the hash value stored in the target slot to obtain a first comparison result;

[0175] 1052b. If the first comparison result is the same, then according to the data item identifier stored in the target slot, a data item is read from the memory container, and the read data item is compared with the target data to obtain a second comparison result. Based on the second comparison result, corresponding processing is performed.

[0176] 1053b. If the first comparison result is different, then the write operation is performed.

[0177] Furthermore, step 1052b above, "performing corresponding processing based on the second comparison result," may include:

[0178] If the second comparison result is the same, then the data duplication processing operation is performed;

[0179] If the second comparison result is different, a new extended block is allocated in the extended linked list area, and the data content stored in the target slot is migrated to the new extended block for storage. At the same time, the offset of the new extended block in the extended linked list area is written to the target slot, so that the target slot becomes an extended pointing slot; and the write operation is performed; wherein, when performing the write operation, the hash value of the target data and the corresponding data item identifier are filled into an empty slot in the new extended block.

[0180] For a detailed description of the implementation of 1051b to 1053b, please refer to other embodiments. Figure 4b The relevant content described.

[0181] In another specific instance, if the target slot is an extended pointer slot, then step 105 above, "based on the type of the target slot, perform corresponding processing operations on the target data," may include:

[0182] 1051c. Perform the jump traversal operation.

[0183] The process of performing corresponding processing based on the traversal result during the jump traversal operation includes:

[0184] If, after traversal, it is determined that the hash value stored in a data slot of the extended block is the same as the hash value of the target data, then data deduplication processing is performed;

[0185] If, after traversal, it is determined that the hash value stored in all data slots of the extended block is different from the hash value of the target data, and an empty slot is encountered, then the write operation is performed; wherein, when performing the write operation, the hash value of the target data and the corresponding data item identifier are filled into the empty slot encountered.

[0186] If, after traversal, it is determined that the hash value stored in all data slots of the extended block is different from the hash value of the target data, and an extended pointing slot is encountered, then the jump traversal operation is returned.

[0187] For a detailed description of the implementation of 1051c here, please refer to other embodiments. Figure 4c The relevant content described.

[0188] The above mainly introduced the operations that the memory manager can perform from the perspective of data writing. The following section introduces the operations that the memory manager can perform from the perspective of data retrieval (data query). Specifically, the data retrieval operation process performed by the memory manager is as follows:

[0189] 41) Calculate the hash value Hg of the data to be searched. For example, after receiving a query request, the hash module 13 can perform a hash calculation on the data to be searched in the query request to obtain the hash value Hg of the data to be searched.

[0190] 42) Based on the slot allocation rules, find a target slot from the base table area (entry part) of the mapping container, and perform different query processing depending on the type of the target slot. Specifically:

[0191] d1. If the target slot is empty, it means that the memory manager does not store the relevant data, so no result was found.

[0192] d2. If the target slot is a data slot, then perform check data processing on the target slot.

[0193] In practice, the data processing can include the following steps:

[0194] First, the hash value Hg can be compared with the hash value stored in the target slot (a data slot) to obtain the first comparison result; then, different processing is performed based on the first comparison result, specifically:

[0195] If the first comparison result is the same, then based on the data item identifier stored in the target slot, the corresponding data item is read from the corresponding storage block in the memory container. Then, the hash value Hg of the data to be found is compared with the hash value and encoded data contained in the data item, or only the encoded data of the data to be found is compared with the encoded data contained in the data item, to obtain the second comparison result. If the second comparison result is the same, the read data item is returned. Specifically, the encoded data contained in the read data item can be decoded to return the decoded data. If the second comparison result is different, no result was found is returned.

[0196] If the first comparison result is different, then no result was found.

[0197] 43) If the target slot is an extended pointer slot, then perform jump traversal processing for the target slot.

[0198] In practice, the jump traversal process may include the following steps:

[0199] Based on the offset of the extended block contained in the extended pointer slot, a jump can be made to the corresponding extended block in the extended linked list area contained in the mapping container, and each slot in the extended block can be traversed sequentially to return the corresponding query result based on the traversal result. Specifically, when traversing to a data slot in the extended block, the data check processing flow described in "d2" above is triggered; when traversing to the extended pointer slot of the extended block, the jump traversal processing is triggered again to jump to the next corresponding extended block to continue the traversal search, and so on until all data is found, and the found data or no query result is returned based on the traversal search result.

[0200] Based on the content here, this application provides another data processing method that the memory manager can perform.

[0201] Figure 6 A flowchart illustrating another data processing method provided in this application is shown. The execution entity of this method is... Figure 1 The memory manager shown contains a mapped container and a memory container. For example... Figure 6 As shown, the data processing method includes:

[0202] 201. In response to the query request, determine the hash value of the requested query data;

[0203] 202. Based on the hash value, find a target slot from the underlying table area contained in the mapping container;

[0204] 203. Execute the corresponding query processing based on the type of the target slot;

[0205] The mapping container further includes an extended linked list area, and both the base table area and the extended blocks in the extended linked list area have multiple slots for storing data. The target slot is one of the multiple slots in the base table area.

[0206] The target slot is of type empty slot, data slot, or extended pointer slot; the empty slot does not contain any data; the data slot contains a hash value and a data item identifier for a data item, the data item identifier indicating the data item corresponding to the data item stored in the memory container, the data item consisting of the hash value and encoded data of the data item, the encoded data being obtained by encoding the data item; the extended pointer slot points to an extended block in the extended linked list area.

[0207] Furthermore, step 203 above, "perform the corresponding query processing according to the type of the target slot," includes:

[0208] If the target slot is empty, return "No results found".

[0209] If the target slot is a data slot, then perform check data processing on the target slot;

[0210] If the target slot is an extended pointer slot, then a jump traversal process is performed for the target slot;

[0211] The aforementioned data processing includes: comparing the hash value with the hash value stored in the data slot to obtain a first comparison result; if the first comparison result is the same, then according to the data item identifier stored in the target slot, reading a corresponding data item from the memory container, and comparing the read data item with the target data to obtain a second comparison result, and returning a corresponding query result based on the second comparison result. Specifically, returning a corresponding query result based on the second comparison result includes: returning the read data item when the second comparison result is the same, and returning no query result when the second comparison result is different.

[0212] Furthermore, the aforementioned jump traversal process includes: jumping to an extension block pointed to by the extension pointer slot, and traversing each slot in the extension block; when traversing to a data slot in the extension block, triggering the execution of the data check process; when traversing to the extension pointer slot in the extension block, triggering the execution of the jump traversal process.

[0213] For details on the implementation of steps 201 to 203 above, please refer to the relevant content in other embodiments.

[0214] It should be noted that any steps in the method provided in this application that are not described in detail can be found in the corresponding content of the above embodiments, and will not be repeated here. Furthermore, the method provided in this application may include other parts or all of the steps in the above embodiments in addition to the steps described above; for details, please refer to the corresponding content of the above embodiments, and will not be repeated here.

[0215] In this application, the memory manager, in addition to the data writing and data retrieval functions described in other embodiments, also has other functions, such as data sorting. Specifically, during sorting, it sorts the data according to a preset sorting rule, based on the decoded hash value or the decoded data obtained by decoding the encoded data in the data item, and outputs the sorting result using the corresponding data item identifier (data ID (Arena item id)).

[0216] For example, the memory container in the memory manager stores the following three data items:

[0217] The data item itme1 contains the hash value Hash:123, the encoded data Value:abc, and the data item identifier (Arena item id) corresponding to the data item item1, which is 0x1000000.

[0218] The data item itme2 contains a hash value Hash:124, encoded data Value:def, and the data item identifier (Arena item id) corresponding to the data item item2 is 0x1000040;

[0219] The data item itme3 contains the hash value Hash:121, the encoded data Value:ppp, and the data item identifier (Arena item id) corresponding to the data item item3 is: 0x1000020;

[0220] The arena item id uniquely identifies a data item stored in the memory container. A data item contains a hash value and a value.

[0221] As mentioned earlier, the arena item ID is returned to the corresponding external service logic module (the caller) so that the external service logic module can perform other processing based on the arena item ID, such as returning it to the front-end user. When the user needs to sort the data, they can directly pass some integer arena item IDs to the memory manager through the front-end page using program code. For example, some arena item IDs passed are 0x1000000, 0x1000040, and 0x1000020. The memory manager will then perform sorting processing on the received data arena item IDs according to the corresponding sorting rules. The sorting rules can be user-defined or default, and are not specifically limited here. For example, if the user selects to sort by lexicographical order of the values, the memory manager will locate the corresponding data item in its containing memory container based on the received data item identifiers. For instance, based on the data item identifier "0x1000000", it will find the data item item1 with offset 0 in memory block 1 of the memory container. Then, decoder 12 will decode the encoded data in data item item1 to obtain the value "abc". This process will continue until all corresponding data items are decoded. Therefore, the final decoded values ​​include "abc", "def", and "ppp". Then, "abc", "def", and "ppp" will be sorted, and the sorting result of the data item identifiers will be output, such as 0x1000000, 0x1000040, and 0x1000020. For example, if the user selects to sort by hash value, the memory manager can find the corresponding hash value in the contained memory container of the received data. For instance, it can find the corresponding data item in the contained memory container, obtain the corresponding hash value from the found data item, use a decoder to decode the found hash value, sort according to the decoded hash value, and output the sorting result corresponding to the data item identifier. For example, the output sorting result is: 0x1000020, 0x1000000, 0x1000040.

[0222] In the solution of this application, data is stored in the memory manager. When data sorting is required, the sorting algorithm can directly call the memory manager and, based on a given number of data item identifiers (Arena item id) and a decoder, directly compare the sizes in the memory manager for sorting. Moreover, no temporary intermediate variables (temporary small variables) need to be generated during sorting, and the sorting result is still output in the form of a data item identifier (Arena item id). It can be seen that the solution of this application reduces the generation of small objects during the sorting process, thereby reducing Java memory allocation and alleviating the pressure on the GC. In traditional Java for sorting complex data types, a custom Comparator interface needs to be defined. For two objects (Object), an integer value needs to be output to indicate the size relationship between the two objects. Among them, the Comparator interface represents a comparator, which is an interface for comparing the sizes of two objects. The Comparator interface defines a compare(To1,To2) method, which returns an integer value based on the sizes of the two objects To1 and To2 to indicate the size relationship between To1 and To2. For example, if the returned integer value is less than 0, it means the comparison result is To1 < To2. In the above traditional Java sorting solution, the sorting implementation process involves the generation of intermediate variables (such as the returned integer value), which increases the pressure on the GC.

[0223] Based on all the content introduced above, a memory manager provided in an embodiment of this application has a structure as Figure 1 and Figure 2 shown. Specifically, the memory manager includes:

[0224] A hash module 13 for performing hash calculation on data to obtain a hash value;

[0225] An encoding module 11 for performing encoding processing on data to obtain encoded data;

[0226] A memory container 16 having multiple storage blocks; the storage blocks are used to store data items, and a data item consists of a hash value of a data and encoded data;

[0227] A mapping container 17 includes a base table area and an extended linked list area. The extended blocks in the base table area and the extended linked list area both have multiple slots for storing data; the types of the slots include empty slots, data slots, and extended pointer slots; no data is stored in the empty slots; a hash value of a data and a data item identifier are stored in the data slots, and the data item identifier can indicate the data item stored in the memory container corresponding to this one data; the extended pointer slot points to an extended block in the extended linked list area;

[0228] The memory manager implements the data processing methods provided in other embodiments of this application through the hash module, the encoding module, the memory container module, and the mapping container.

[0229] In addition to the modules mentioned above, the memory manager provided in this embodiment also includes other modules, such as decoding module 12, comparison module 13, and pointer allocation module 15. For a detailed description of the functions of each module included in the memory manager, please refer to the relevant content in other embodiments.

[0230] The aforementioned memory manager can be applied in a storage engine, which can be, but is not limited to, a columnar storage engine. Therefore, this application also provides a storage engine that includes the memory manager provided in this application.

[0231] Furthermore, the storage engine can be deployed on an electronic device, such as a storage server. A storage server can be one or more servers. It can be a physical server or a virtual server, etc. The storage server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Therefore, one embodiment of this application also provides an electronic device that includes the storage engine provided in this application.

[0232] In addition, the electronic device may include other components, such as memory and a processor. The memory may be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Electrically Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Memory 1101 is used to store a program; a processor, coupled to the memory 1101, is used to execute the program stored in the memory to implement the methods provided in the above-described method embodiments through a storage engine. Furthermore, the electronic device also includes other components such as communication components and power supply components. The above is merely an illustrative representation of some of the components included in an electronic device, and does not imply that the electronic device includes only the aforementioned components.

[0233] Accordingly, one embodiment of this application also provides a database program product, which includes the storage engine provided in other embodiments of this application. A database program product refers to a software product used for managing, storing, and retrieving data.

[0234] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the methods provided in the above-described method embodiments.

[0235] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps or functions of the methods provided in the above-described method embodiments.

[0236] In addition to the above, this application also provides apparatus corresponding to other method embodiments. Specifically, as follows:

[0237] Figure 7 A schematic diagram of the structure of a data processing apparatus provided in this application is shown. The apparatus is deployed in… Figure 1 The memory manager shown contains memory containers with multiple storage blocks. For example... Figure 7 The data processing device includes:

[0238] The hash module is used to perform hash calculations on target data to obtain hash values;

[0239] An encoding module is used to encode the target data to obtain encoded data; the encoding method is variable-length encoding.

[0240] The writing module is used to write the hash value and the encoded data into a data item and then into a target storage block; wherein the target storage block is one of the multiple storage blocks contained in the memory container.

[0241] Furthermore, when the above-mentioned writing module is used to write the hash value and the encoded data into the target storage block, it is specifically used to: determine the target storage block and the write start offset of the target storage block according to the current pointing information of the allocation pointer in the memory manager; using the write start offset of the target storage block as the writing start point, write the hash value and the encoded data into the target storage block as a data item; after the data item is written, update the pointing information of the allocation pointer so that the write start offset pointed to by the allocation pointer is updated to the end position of the data item writing.

[0242] Further, the pointing information includes the first storage block currently pointed to by the allocation pointer and the write start offset of the first storage block; and, when the above-mentioned writing module determines the target storage block and the write start offset of the target storage block according to the current pointing information of the allocation pointer, it can be specifically used to: determine whether the remaining space of the first storage block meets the writing requirements of the target data; if it does, then the first storage block is determined as the target storage block, and the write start offset of the first storage block is determined as the write start offset of the target storage block; if it does not meet, then a free second storage block is allocated by the allocation pointer and determined as the target storage block, and the beginning position of the second storage block is the write start offset of the target storage block; wherein, the first storage block and the second storage block are storage blocks among the plurality of storage blocks.

[0243] Furthermore, the memory manager also includes a mapping container with a base table area and an extended linked list area. Each extended block in the base table area and the extended linked list area has multiple slots for storing data. The device further includes a lookup module and an execution module. The lookup module is used to find a target slot in the base table area based on the hash value. The execution module is used to perform corresponding processing operations on the target data based on the type of the target slot. The target slot is of type empty slot, data slot, or extended pointer slot; the empty slot does not contain data; the data slot stores a hash value and a data item identifier for a data item, the data item identifier indicating the data item corresponding to that data item stored in the memory container; the extended pointer slot points to an extended block in the extended linked list area. The processing operations include at least one of a write operation, a data repetition operation, and a jump traversal operation. The write operation includes: writing a data item consisting of the hash value and the encoded data corresponding to the target data into the target storage block; determining the data item identifier corresponding to the target data based on the block identifier of the target storage block and the write start offset; and filling the hash value and the data item identifier into a corresponding slot in the mapping container. The data deduplication operation includes outputting error information or data replacement processing; the data replacement processing includes replacing a data item in the memory container indicated by the data item identifier stored in the data slot with a data item consisting of the hash value and encoded data of the target data. The jump traversal operation includes: jumping to the extended block pointed to by the extended pointer slot, and sequentially traversing each slot in the extended block to perform corresponding processing based on the traversal results.

[0244] Furthermore, if the target slot is an empty slot, the execution module, when performing corresponding processing operations on the target data based on the type of the target slot, can specifically be used to: perform the write operation; wherein, when performing the write operation, the hash value of the target data and the corresponding data item identifier are filled into the target slot.

[0245] Furthermore, if the target slot is a data slot, the execution module, when performing corresponding processing operations on the target data based on the type of the target slot, can specifically be used to: compare the hash value of the target data with the hash value stored in the target slot to obtain a first comparison result; if the first comparison result is the same, then according to the data item identifier stored in the target slot, read a data item from the memory container, and compare the read data item with the target data to obtain a second comparison result, and perform corresponding processing based on the second comparison result; if the first comparison result is different, then perform the write operation.

[0246] The process of performing corresponding processing based on the second comparison result includes: if the second comparison result is the same, then performing the data duplication processing operation; if the second comparison result is different, then allocating a new extension block in the extended linked list area, migrating the data content stored in the target slot to the new extension block for storage, and simultaneously writing the offset of the new extension block in the extended linked list area to the target slot, so that the target slot becomes an extended pointing slot; and performing the write operation; wherein, when performing the write operation, the hash value of the target data and the corresponding data item identifier are filled into an empty slot in the new extension block.

[0247] Further, if the target slot is an extended pointer slot, then when the above-mentioned execution module performs corresponding processing operations on the target data based on the type of the target slot, it can specifically be used to: execute the jump traversal operation; wherein, when performing the corresponding processing based on the traversal result in the jump traversal operation, it includes: if the traversal determines that the hash value stored in a data slot in the extended block is the same as the hash value of the target data, then perform data duplication processing; if the traversal determines that the hash values ​​stored in all data slots in the extended block are different from the hash value of the target data, and an empty slot is encountered, then perform the write operation; wherein, when performing the write operation, the hash value of the target data and the corresponding data item identifier are filled into the empty slot encountered; if the traversal determines that the hash values ​​stored in all data slots in the extended block are different from the hash value of the target data, and an extended pointer slot is encountered, then return to execute the jump traversal operation.

[0248] For details regarding the specific implementation of the functions of each module in the device provided in this embodiment, please refer to the relevant content in other embodiments of this application. Furthermore, in addition to the functional steps described above, each module in the device provided in this embodiment can also implement other functional steps. For details regarding these other functional steps, please refer to the relevant content in other embodiments of this application.

[0249] Figure 8 A schematic diagram of another data processing apparatus provided in this application is shown. This apparatus is deployed in… Figure 1 The memory manager shown contains a mapping container and a memory container. For example... Figure 8 The data processing device includes:

[0250] The determination module is used to determine the hash value of the requested query data in response to a query request;

[0251] A lookup module is used to find a target slot from the underlying table area contained in the mapping container based on the hash value;

[0252] The execution module is used to perform corresponding query processing based on the type of the target slot;

[0253] The mapping container further includes an extended linked list area, and both the base table area and the extended blocks in the extended linked list area have multiple slots for storing data. The target slot is one of the multiple slots in the base table area.

[0254] The target slot is of type empty slot, data slot, or extended pointer slot; the empty slot does not contain any data; the data slot contains a hash value and a data item identifier for a data item, the data item identifier indicating the data item corresponding to the data item stored in the memory container, the data item consisting of the hash value and encoded data of the data item, the encoded data being obtained by encoding the data item; the extended pointer slot points to an extended block in the extended linked list area.

[0255] Furthermore, the aforementioned execution module, when performing corresponding query processing based on the type of the target slot, can specifically be used for: if the target slot is an empty slot, returning no query result; if the target slot is a data slot, performing check data processing for the target slot; if the target slot is an extended pointer slot, performing jump traversal processing for the target slot. The check data processing includes: comparing the hash value with the hash value stored in the data slot to obtain a first comparison result; if the first comparison result is the same, then according to the data item identifier stored in the target slot, reading a corresponding data item from the memory container, comparing the read data item with the target data to obtain a second comparison result, and returning the corresponding query result based on the second comparison result. The jump traversal process includes: jumping to an extension block pointed to by the extension pointer slot, and traversing each slot in the extension block; when traversing to a data slot in the extension block, triggering the execution of the data check process; when traversing to the extension pointer slot in the extension block, triggering the execution of the jump traversal process.

[0256] For details regarding the specific implementation of the functions of each module in the device provided in this embodiment, please refer to the relevant content in other embodiments of this application. Furthermore, in addition to the functional steps described above, each module in the device provided in this embodiment can also implement other functional steps. For details regarding these other functional steps, please refer to the relevant content in other embodiments of this application.

[0257] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0258] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM (Read Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0259] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, The application is applied to a memory manager comprising a memory container, wherein the memory container comprises a plurality of memory blocks; the method comprises: hashing target data to obtain a hash value; encoding the target data to obtain encoded data, wherein the encoding mode is variable-length encoding; writing the hash value and the encoded data into a target memory block, wherein the target memory block is one of the plurality of memory blocks comprised by the memory container.

2. The method of claim 1, wherein, The writing of the hash value and the encoded data into the target memory block comprises: determining the target memory block and a write start offset of the target memory block according to current pointing information of an allocation pointer in the memory manager; writing the hash value and the encoded data into the target memory block as a data item starting from the write start offset of the target memory block; after the writing of the data item is completed, updating the pointing information of the allocation pointer so that the write start offset pointed to by the allocation pointer is updated to the end position of the writing of the data item.

3. The method of claim 2, wherein, The pointing information comprises a first memory block currently pointed to by the allocation pointer and a write start offset of the first memory block; and determining the target memory block and the write start offset of the target memory block according to the current pointing information of the allocation pointer comprises: determining whether a remaining space of the first memory block meets the writing requirement of the target data; if yes, determining the first memory block as the target memory block and determining the write start offset of the first memory block as the write start offset of the target memory block; if no, determining an idle second memory block allocated by the allocation pointer as the target memory block, and the beginning position of the second memory block as the write start offset of the target memory block; wherein the first memory block and the second memory block are memory blocks in the plurality of memory blocks. The memory manager further comprises a mapping container comprising a base table area and an extended chain table area, wherein each of the extended blocks in the base table area and the extended chain table area comprises a plurality of slot positions for storing data; and the mapping container further comprises:

4. The method according to any one of claims 1 to 3, characterized in that, finding a target slot position from the base table area according to the hash value; performing a corresponding processing operation on the target data based on the type of the target slot position; wherein the type of the target slot position is an empty slot position, a data slot position or an extended pointing slot position; the empty slot position does not store data; the data slot position stores a hash value of one data and a data item identifier, wherein the data item identifier can indicate a data item corresponding to the one data stored in the memory container; and the extended pointing slot position points to an extended block in the extended chain table area; the processing operation comprises at least one of a writing operation, a data repetition processing operation and a jump traversal operation. ​ ​ The write operation comprises: writing the hash value corresponding to the target data and the encoded data into a data item in the target storage block, and determining a data item identifier corresponding to the target data according to a block identifier of the target storage block and the write start offset, and filling the hash value and the data item identifier into a corresponding slot in the mapping container; The data duplication processing operation comprises outputting error information or data replacement processing; the data replacement processing comprises replacing a data item indicated by a data item identifier stored in a data slot in the memory container with a data item composed of the hash value of the target data and the encoded data; The jump traversal operation comprises: jumping to an extension block pointed to by an extension pointing slot, and sequentially traversing each slot in the extension block to perform corresponding processing according to the traversal result.

5. The method of claim 4, wherein, If the target slot is an empty slot, then: Based on the type of the target slot, a corresponding processing operation is performed on the target data, including: performing the write operation; wherein, in the write operation, the hash value of the target data and the corresponding data item identifier are filled into the target slot.

6. The method of claim 4, wherein, If the target slot is a data slot, then: Based on the type of the target slot, a corresponding processing operation is performed on the target data, including: comparing the hash value of the target data with the hash value stored in the target slot to obtain a first comparison result; if the first comparison result is the same, then a data item is read from the memory container according to the data item identifier stored in the target slot, and the read data item is compared with the target data to obtain a second comparison result, and a corresponding processing is performed based on the second comparison result; if the first comparison result is different, then the write operation is performed.

7. The method of claim 6, wherein, Based on the second comparison result, a corresponding processing is performed, including: if the second comparison result is the same, then the data duplication processing operation is performed; if the second comparison result is different, then a new extension block is allocated in the extension chain table area, and the data content stored in the target slot is migrated to the new extension block for storage, and the offset of the new extension block in the extension chain table area is written to the target slot, so that the target slot is changed to an extension pointing slot; and the write operation is performed; wherein, in the write operation, the hash value of the target data and the corresponding data item identifier are filled into an empty slot in the new extension block.

8. The method of claim 4, wherein, If the target slot is an extension pointing slot, then: Based on the type of the target slot, a corresponding processing operation is performed on the target data, including: performing the jump traversal operation; wherein, in the jump traversal operation, when a corresponding processing is performed according to the traversal result, it includes: if it is determined through traversal that the hash value stored in a data slot in the extension block is the same as the hash value of the target data, then data duplication processing is performed. If it is determined through the traversal that the hash value stored in all data slot positions in the extension block is not the same as the hash value of the target data, and an empty slot position is encountered, the write operation is performed; wherein, in performing the write operation, the hash value of the target data and the corresponding data item identifier are written into the encountered empty slot position; If it is determined through the traversal that the hash value stored in all data slot positions in the extension block is not the same as the hash value of the target data, and an extension pointing slot position is encountered, the jump traversal operation is returned to be performed.

9. A data processing method, characterized by, The method is applied to a memory manager comprising a mapping container and a memory container; the method comprises: In response to a query request, determining a hash value of requested query data; According to the hash value, finding a target slot position from a basic table area contained in the mapping container; According to the type of the target slot position, performing corresponding query processing; The mapping container further comprises an extension chain table area, and each of the extension block in the basic table area and the extension chain table area has a plurality of slot positions for storing data, and the target slot position is one of the plurality of slot positions in the basic table area; The type of the target slot position is an empty slot position, a data slot position, or an extension pointing slot position; the empty slot position does not store data; the data slot position stores a hash value of one data and a data item identifier, the data item identifier can indicate a data item corresponding to the one data stored in the memory container, the data item is composed of the hash value of the one data and encoded data, and the encoded data is obtained by encoding the one data; the extension pointing slot position points to an extension block in the extension chain table area.

10. The method of claim 9, wherein, According to the type of the target slot position, performing corresponding query processing, comprising: If the target slot position is an empty slot position, return that no result is queried; If the target slot position is a data slot position, perform check data processing on the target slot position; If the target slot position is an extension pointing slot position, perform jump traversal processing on the target slot position; The check data processing comprises: comparing the hash value with the hash value stored in the data slot position to obtain a first comparison result; if the first comparison result is the same, reading a corresponding data item from the memory container according to the data item identifier stored in the target slot position, and comparing the read data item with the target data to obtain a second comparison result, and returning corresponding query results based on the second comparison result; The jump traversal processing comprises: jumping to an extension block pointed by the extension pointing slot position, and traversing each slot position in the extension block; when a data slot position in the extension block is traversed, the check data processing is triggered to be performed; when an extension pointing slot position in the extension block is traversed, the jump traversal processing is triggered to be performed.

11. A memory manager, characterized by Comprise: A hash module for performing hash calculation on data to obtain a hash value; An encoding module for performing encoding processing on data to obtain encoded data; A memory container having a plurality of storage blocks; The storage block is used for storing data items, and one data item is composed of a hash value of one data and encoded data; A mapping container, comprising a basic table area and an extended chain table area, each of the basic table area and the extended chain table area having a plurality of slot positions for storing data; types of the slot positions including an empty slot position, a data slot position, and an extended pointing slot position; no data being stored in the empty slot position; a hash value of one data and a data item identifier being stored in the data slot position, the data item identifier being able to indicate a data item corresponding to the one data stored in the memory container; the extended pointing slot position pointing to an extended block in the extended chain table area; the memory manager realizing the data processing method of any one of claims 1 to 10 through the hash module, the encoding module, the memory container module, and the mapping container.

12. A storage engine, comprising: The memory manager of claim 11.

13. An electronic device, comprising: The storage engine of claim 12.

14. A database program product, characterized by The storage engine of claim 12.

15. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by a computer to realize the method of any one of claims 1 to 10.

16. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to realize the method of any one of claims 1 to 10.

Images