Data storage method, data query method, device and electronic device
By maintaining an array of index file position information of hash values and keys, as well as an array of value, and building an index file to store and query data, it solves the problem of excessive memory consumption of HashMap when storing massive data, and improves query efficiency.
Patent Information
- Application Number
- CN202111637486.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-29
AI Technical Summary
When HashMap in Java stores massive key-value pair data, memory space consumption is too large due to memory loading related object references and internal data structures, and lacks efficient storage and query methods.
By maintaining the first array of hash values and key location information in the index file, and the second array storing value, the index file is built to store the location of keys and value, reducing memory space consumption, and quickly locate data during querying , avoid full data loading.
It effectively reduces the consumption of memory space and improves the efficiency of data query, especially when processing massive data, which significantly saves memory resources.
Smart Images

Figure CN114265787B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data storage method, a data query method, a device and an electronic device. Background Art
[0002] Java is one of the most popular programming languages for software system development, with a huge number of users. Java's HashMap is a key-value pair collection implemented using a hash table. HashMap can load and retrieve massive key-value data, but when storing massive key-value data, it involves memory loading of basic type values, such as related object references and actual occupancy of some objects in the internal data structure, which will lead to a huge consumption of memory space. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a data storage method, a data query method, a device and an electronic device, so as to improve the problem of large memory space consumption caused by the method of storing data in a hash table in the prior art.
[0004] In a first aspect, an embodiment of the present application provides a data storage method, the method comprising:
[0005] Acquire the data to be stored, where the data to be stored is of key-value type;
[0006] Calculate and obtain the hash value to be stored corresponding to the key to be stored in the data to be stored;
[0007] According to the hash value to be stored, the key to be stored is stored at a corresponding position in the index file of the disk, and the position information of the corresponding position is obtained;
[0008] Storing the hash value to be stored and the location information in a first array in a memory;
[0009] The value corresponding to the key to be stored is stored in a second array in the memory, and the element position where the value corresponding to the key to be stored is located in the second array is stored at the position corresponding to the key to be stored in the index file.
[0010] In the above implementation process, by maintaining a first array storing hash values and key position information in an index file, and maintaining a second array storing value, and building an index file to store the positions of key and value in the second array, only the first array and the second array need to be maintained in the memory, and the index file is stored on the disk. Compared with the existing method of using a hash table to store data, this solution can effectively reduce the consumption of memory space.
[0011] Optionally, storing the value corresponding to the key to be stored in a second array in the memory includes:
[0012] Determine whether there is a value in the second array of the memory that is the same as the value corresponding to the key to be stored;
[0013] If so, replace the same value with the value corresponding to the key to be stored and store it in the second array.
[0014] In the above implementation process, the values stored in the second array are deduplicated, which can further save memory space.
[0015] Optionally, storing the key to be stored at a corresponding position in an index file on a disk according to the hash value to be stored includes:
[0016] According to the hash value to be stored, the key to be stored is stored in the row corresponding to the hash value to be stored in the index file of the disk;
[0017] The obtaining the location information of the corresponding location includes:
[0018] Gets the byte offset of the line.
[0019] In the above implementation process, the key in the index file is located by the byte offset of the row, so that when querying data, the position of the row to be searched can be quickly located, thereby improving the search efficiency.
[0020] Optionally, the element position where the value corresponding to the key to be stored is located is the array element offset corresponding to the value, and the length of the array element offset is consistent with the maximum length of the elements in the second array. In this way, when searching for data, fixed-length data can be directly skipped for searching, thereby improving search efficiency.
[0021] Optionally, storing the element position where the value corresponding to the key to be stored in the second array is located at a position corresponding to the key to be stored in the index file includes:
[0022] If the length of the array element offset corresponding to the value corresponding to the key to be stored is less than the maximum length, the array element offset is padded with zeros to make the length of the array element offset equal to the maximum length;
[0023] The zero-filled array element offset is stored in the index file at a position corresponding to the key to be stored.
[0024] In the above implementation process, the length of the array element offset is set to the maximum length through the zero padding operation, so that the lengths of all array element offsets can be consistent without affecting the value of the array element offset.
[0025] In a second aspect, an embodiment of the present application provides a data query method, the method comprising:
[0026] Obtain the target data to be queried, where the target data is key-value type data;
[0027] Calculate and obtain the target hash value corresponding to the target key in the target data;
[0028] Searching and obtaining the target file position corresponding to the target hash value from a first array in memory, wherein each element in the first array is used to store the hash value of the key and the corresponding file position, the file position is used to indicate the position of the key in the index file of the disk, the index file also stores the element position of the value corresponding to the key in a second array in memory, and each element in the second array is used to store the value;
[0029] Search the index file according to the target file position to obtain the element position of the value corresponding to the target key in the second array;
[0030] According to the element position of the value corresponding to the target key in the second array, the value corresponding to the target key is obtained from the second array.
[0031] In the above implementation process, when performing data query, by searching the first array and the second array in the memory, and searching the information of the target file location in the index file, the problem of loading the full amount of data into the memory can be avoided, the consumption of memory space can be reduced, and the query efficiency can be improved.
[0032] In a third aspect, an embodiment of the present application provides a data storage device, the device comprising:
[0033] A data acquisition module is used to acquire data to be stored, wherein the data to be stored is key-value type data;
[0034] A hash calculation module, used to calculate and obtain a hash value to be stored corresponding to a key to be stored in the data to be stored;
[0035] A key storage module, used to store the key to be stored at a corresponding position in the index file of the disk according to the hash value to be stored, and obtain position information of the corresponding position;
[0036] A hash value storage module, used for storing the hash value to be stored and the position information in a first array of a memory;
[0037] The value storage module is used to store the value corresponding to the key to be stored in the second array of the memory, and store the element position of the value corresponding to the key to be stored in the second array at the position corresponding to the key to be stored in the index file.
[0038] In a fourth aspect, an embodiment of the present application provides a data query device, the device comprising:
[0039] A query data acquisition module is used to acquire target data to be queried, where the target data is key-value type data;
[0040] A data hash calculation module, used to calculate and obtain a target hash value corresponding to a target key in the target data;
[0041] A file location search module is used to search for a target file location corresponding to the target hash value from a first array in memory, wherein each element in the first array is used to store the hash value of the key and the corresponding file location, the file location is used to indicate the location of the key in the index file of the disk, the index file also stores the element location of the value corresponding to the key in a second array in memory, and each element in the second array is used to store the value;
[0042] An element position search module, used to search the index file according to the target file position to obtain the element position of the value corresponding to the target key in the second array;
[0043] The value search module is used to obtain the value corresponding to the target key from the second array according to the element position of the value corresponding to the target key in the second array.
[0044] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect or the second aspect are executed.
[0045] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method provided in the first aspect or the second aspect are performed.
[0046] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 A flowchart of a data storage method provided in an embodiment of the present application;
[0049] Figure 2 A flowchart of a data query method provided in an embodiment of the present application;
[0050] Figure 3 A structural block diagram of a data storage device provided in an embodiment of the present application;
[0051] Figure 4 A structural block diagram of a data query device provided in an embodiment of the present application;
[0052] Figure 5 A schematic diagram of the structure of an electronic device for executing a data storage method or a data query method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application.
[0054] It should be noted that the terms "system" and "network" in the embodiments of the present invention can be used interchangeably. "Multiple" means two or more. In view of this, in the embodiments of the present invention, "multiple" can also be understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the previously associated objects are in an "or" relationship.
[0055] An embodiment of the present application provides a data storage method, which maintains a first array storing hash values and key position information in an index file, and maintains a second array storing value, and stores the positions of key and value in the second array by constructing an index file. In this way, only the first array and the second array need to be maintained in the memory, and the index file is stored on the disk. Compared with the existing method of using a hash table to store data, this solution can effectively reduce memory usage.
[0056] Please refer to Figure 1 , Figure 1 A data storage method provided in an embodiment of the present application comprises the following steps:
[0057] Step S110: Acquire the data to be stored, where the data to be stored is of key-value type.
[0058] The data to be stored may refer to data that needs to be stored currently, which may be one piece of data or multiple pieces of data, and the storage of each piece of data may be performed according to the storage method of the present application. The data to be stored in the present application may be understood as one of the pieces of data that needs to be stored.
[0059] Step S120: Calculate and obtain the hash value to be stored corresponding to the key to be stored in the data to be stored.
[0060] The hash value to be stored here can be understood as the result of performing a hash modulo operation on the key to be stored through a hash function. For example, if the total number of data to be stored is n, the remainder obtained by dividing the value of the key to be stored by n is the hash value to be stored. Of course, the total number of data n here can be flexibly set according to actual needs.
[0061] Step S130: According to the hash value to be stored, the key to be stored is stored at a corresponding position in the index file of the disk, and the position information of the corresponding position is obtained.
[0062] The index file is pre-built and stored on the disk. The key to be stored is stored in the index file. The hash value to be stored is used to indicate that the key to be stored is stored at the corresponding position of the index file. For example, if the hash value to be stored is used to indicate a row of the index file, the key to be stored can be stored in the row of the index file corresponding to the hash value to be stored according to the hash value to be stored, and then the location information of the row can be obtained.
[0063] For example, if the hash value to be stored is 2, it means that the key to be stored can be stored in the second line of the index file. If there is no data in the second line of the current index file, the key to be stored can be stored at the first data position of the second line. If data is already stored in the second line of the current index file, the key to be stored can be stored after the stored data in the second line.
[0064] It is understandable that the hash value to be stored can also be used to indicate a column in the index file, that is, the key to be stored can also be stored in a column in the index file according to the hash value to be stored. Of course, the hash value to be stored can also be used to indicate other parameters that can represent positions in the index file.
[0065] When the hash value to be stored is used to indicate the row position in the index file, the position information corresponding to the row position can be a byte offset, such as each row has a corresponding byte offset, and the byte offset is used to indicate the row position in the index file, such as the byte offset of the first row is 0, the byte offset of the second row is 3000, and the byte offset of the third row is 5000. It can be understood that when the corresponding hash value to be stored is used to indicate the column position of the index file, the column position can also be indicated by a byte offset.
[0066] Step S140: storing the hash value and location information to be stored in a first array in the memory.
[0067] After obtaining the hash value and location information to be stored, the two information can be stored in an element of the first array. The first array can also be pre-built and stored in the memory, and each element therein is used to store the hash value and location information. When storing, the hash value to be stored can be used as the subscript of the numerical element, and the value of the array element is used to store the byte offset. For example, an element a[2]=3000 in the first array, where 2 represents the hash value to be stored and 3000 represents the location information, such as the byte offset of the above row position.
[0068] The total length of the first array is the total number of data items n, and the type of the first array is Integer.
[0069] Step S150: Store the value corresponding to the key to be stored in the second array in the memory, and store the element position of the value corresponding to the key to be stored in the second array at the position corresponding to the key to be stored in the index file.
[0070] The value in the data to be stored can be stored in the constructed second array, that is, each element in the second array is used to store the value. When storing the value of the data to be stored, it can be directly stored in the newly created array element after the data stored in the second array.
[0071] The position of the element in the second array where the value is stored can then be stored in the index file.
[0072] For example, the value of the data to be stored is stored in the second array element in the second array, and the key to be stored is stored in the second row and second column of the index file (such as the byte offset is 3000). Then the position of the second array element in the second array (such as position 2) can also be stored in the second row and second column of the index file. The two pieces of information stored in the index file can be expressed as: key to be stored - 2. The hash value and byte offset to be stored are stored in the first array, such as a[2] = 3000.
[0073] Therefore, the index file stores the position index of the key to be stored and the corresponding value. In this way, when searching for data, the byte offset of the index file can be found through the first array, and then the position index of the value can be found from the corresponding position of the index file according to the byte offset, and then the corresponding value can be read from the second array according to the position index. In this way, there is no need to load all the data in the index file into the memory, which can reduce memory consumption and improve search efficiency.
[0074] In the above implementation process, by maintaining a first array storing hash values and key position information in an index file, and maintaining a second array storing value, and building an index file to store the positions of key and value in the second array, only the first array and the second array need to be maintained in the memory, and the index file is stored on the disk. Compared with the existing method of using a hash table to store data, this solution can effectively reduce the consumption of memory space.
[0075] On the basis of the above embodiment, in order to further reduce the occupation of memory resources, when storing the value corresponding to the key to be stored in the second array, it is also possible to first determine whether there is a value in the second array that is identical to the value corresponding to the key to be stored. If so, replace the value that is identical to the value corresponding to the key to be stored with the value corresponding to the key to be stored and store it in the second array.
[0076] That is to say, when storing values, duplicates can be removed. For example, if the value corresponding to the key to be stored is "zhangsan", it is found through judgment that the same value "zhangsan" has been stored in the second array. In order to avoid the problem of storing duplicate data and occupying storage space, the stored value can be directly replaced with the current value to be stored. Alternatively, the stored value can be retained without replacement, and the current value to be stored can be discarded.
[0077] Of course, if you do not want to replace it, you can also directly store the value to be stored after the stored data.
[0078] In the above implementation process, the values stored in the second array are deduplicated to avoid storing duplicate values, which can further save memory space.
[0079] Based on the above embodiment, the element position where the value corresponding to the key to be stored is located is the array element offset corresponding to the value, and the length of the array element offset is consistent with the maximum length of the elements in the second array.
[0080] For example, if you want to store 1,000 pieces of data, and the length of the value in each piece of data is inconsistent, in order to facilitate subsequent searches by length, you can pre-count the maximum length of the elements in the second array. If the maximum length is "145", it occupies 3 bytes, and the value to be stored needs to be stored in the 9th element, indicating that its numerical element offset is 9. At this time, its array element offset occupies 1 byte and the length is inconsistent. In order to facilitate subsequent searches for data through index files according to a certain rule, the length of the array element offset can be kept consistent with the maximum length.
[0081] The specific approach is that if the length of the array element offset corresponding to the value corresponding to the key to be stored is less than the maximum length, the array element offset is padded with zeros to make the length of the array element offset equal to the maximum length, and then the zero-padded array element offset is stored in the position corresponding to the key to be stored in the index file.
[0082] For example, if the array element offset is 9, you can perform a zero padding operation to ensure that the length of the filled offset is consistent with the maximum length. For example, the array element offset after the zero padding operation is "009", and the array element offset after the zero padding occupies 3 bytes. For other array element offsets whose length is less than the maximum length, you can also perform zero padding at the highest bit according to the same operation.
[0083] In the above implementation process, the length of the array element offset is set to the maximum length through the zero padding operation, so that the lengths of all array element offsets can be consistent without affecting the value of the array element offset.
[0084] On the basis of the above embodiment, in order to facilitate the search of index files, the FileChannel map method in the Java API can also be used to map the index file to a memory image file. This step does not occupy memory. In this way, when searching for data, the corresponding position of the index file can be located according to the memory image file, and the API can be used to quickly jump to the relevant position of the index file and obtain the array element offset of the value.
[0085] There are two situations for the array element offset of the obtained value:
[0086] 1. When there is no conflict in hash values (i.e., there is only one hash value in the first array that is the same as the hash value to be searched), since the lengths of the compared key values are consistent, the array element offset of the value can be obtained by directly jumping a certain length.
[0087] 2. In the case of hash value conflicts (i.e., there are more than two hash values in the first array that are the same as the hash value to be searched), at this time, more than two byte offsets of the index file can be obtained from the first array. If the key is matched at the corresponding position according to the first byte offset, if the first key value does not match after comparison, since the byte offset has been determined, the position where the second byte offset is located can be directly skipped to match the key.
[0088] After matching the key, you can get the array element offset, and then you can search the second array for the corresponding value based on the array element offset.
[0089] Therefore, the above data storage method can greatly reduce memory usage. For example, in the query of domain name classification, there are about 10 million data in the provided domain name classification file, such as "0000006c574aa2|traditional industry|chemical industry". The data query logic is to obtain query results such as "traditional industry, chemical industry" and other information according to the key value "0000006c574aa2". In the domain name classification, the original file is about 300M. If all the data results are loaded into the memory using HashMap, it will take up about 5G of memory space. If the data storage method of this application is used, about 4.5G of memory space can be saved.
[0090] The following describes the data query process, please refer to Figure 2 , Figure 2 A flowchart of a data query method provided in an embodiment of the present application, the method comprising the following steps:
[0091] Step S210: Acquire target data to be queried, where the target data is key-value type data.
[0092] Since the data is stored in key-value type, if the target data to be queried is not key-value type data, it needs to be converted to data of this type before querying. The target data can refer to one of the data to be queried, and the query process for each data can be queried according to the data query method in this application. Here, one of the data is used as the target data for querying as an example.
[0093] Step S220: Calculate and obtain the target hash value corresponding to the target key in the target data.
[0094] The target hash value is calculated in the same manner as the above-mentioned method of calculating the hash value to be stored corresponding to the key to be stored, such as dividing the value of the target key by the total number of stored data items n to obtain the remainder as the target hash value.
[0095] Step S230: searching the first array in the memory to obtain the target file location corresponding to the target hash value.
[0096] Among them, please refer to the relevant description in the above embodiment, each element in the first array is used to store the hash value of the key and the corresponding file position, the file position is used to indicate the location of the key in the index file of the disk, and the index file also stores the element position of the value corresponding to the key in the second array, and each element in the second array is used to store the value.
[0097] Therefore, after obtaining the target hash value, the target hash value can be matched with each hash value in the first array. If a matching hash value is found, the file position corresponding to the matching hash value can be obtained as the target file position. The file position can be the byte offset of the row position in the index file, that is, the target key is stored in the row corresponding to the byte offset.
[0098] Step S240: Search the index file according to the target file position to obtain the element position of the value corresponding to the target key in the second array.
[0099] After obtaining the byte offset, you can locate the row position corresponding to the byte offset in the index file, and then match the target key with each key stored in the row to find the matching key. Since the matching key also stores the element position of the value corresponding to the key in the second array in the memory, you can obtain the element position of the value corresponding to the target key in the second array through the index file.
[0100] Step S250: Obtain the value corresponding to the target key from the second array according to the element position of the value corresponding to the target key in the second array.
[0101] The element position is the array element offset described in the above embodiment, so the value corresponding to the target key can be found from the second array according to the element position, thereby completing the data search.
[0102] It should be noted that the storage method of the data in the first array, the second array and the index file involved in the data query method can refer to the corresponding process of the data storage method in the above embodiment, and for the sake of brevity of description, it will not be elaborated here.
[0103] In the above implementation process, when performing data query, by searching the first array and the second array in the memory, and searching the information of the target file location in the index file, the problem of loading the full amount of data into the memory can be avoided, the consumption of memory space can be reduced, and the query efficiency can be improved.
[0104] Please refer to Figure 3 , Figure 3 This is a structural block diagram of a data storage device 300 provided in an embodiment of the present application. The device 300 may be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device 300 can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here.
[0105] Optionally, the device 300 includes:
[0106] The data acquisition module 310 is used to acquire the data to be stored, where the data to be stored is of key-value type;
[0107] A hash calculation module 320 is used to calculate and obtain a hash value to be stored corresponding to a key to be stored in the data to be stored;
[0108] The key storage module 330 is used to store the key to be stored at a corresponding position in the index file of the disk according to the hash value to be stored, and obtain the position information of the corresponding position;
[0109] A hash value storage module 340, configured to store the hash value to be stored and the location information in a first array of a memory;
[0110] The value storage module 350 is used to store the value corresponding to the key to be stored in the second array of the memory, and store the element position of the value corresponding to the key to be stored in the second array at the position corresponding to the key to be stored in the index file.
[0111] Optionally, the value storage module 350 is used to determine whether there is a value in the second array of the memory that is identical to the value corresponding to the key to be stored; if so, replace the identical value with the value corresponding to the key to be stored and store it in the second array.
[0112] Optionally, the key storage module 330 is used to store the key to be stored in a row corresponding to the hash value to be stored in the index file according to the hash value to be stored; and obtain a byte offset of the row.
[0113] Optionally, the element position where the value corresponding to the key to be stored is located is the array element offset corresponding to the value, and the length of the array element offset is consistent with the maximum length of the elements in the second array.
[0114] Optionally, the value storage module 350 is used to fill the array element offset with zeros if the length of the array element offset corresponding to the value corresponding to the key to be stored is less than the maximum length, so that the length of the array element offset is equal to the maximum length; and store the zero-filled array element offset at the position corresponding to the key to be stored in the index file.
[0115] Please refer to Figure 4 , Figure 4 The structure block diagram of a data query device 400 provided in an embodiment of the present application, the device 400 may be a module, program segment or code on an electronic device. It should be understood that the device 400 is similar to the above Figure 2 The method embodiment corresponds to and can be executed Figure 2 The various steps involved in the method embodiment and the specific functions of the device 400 can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here.
[0116] Optionally, the device 400 includes:
[0117] The query data acquisition module 410 is used to acquire the target data to be queried, where the target data is key-value type data;
[0118] A data hash calculation module 420 is used to calculate and obtain a target hash value corresponding to a target key in the target data;
[0119] A file location search module 430 is used to search for a target file location corresponding to the target hash value from a first array in memory, wherein each element in the first array is used to store the hash value of the key and the corresponding file location, the file location is used to indicate the location of the key in the index file of the disk, the index file also stores the element location of the value corresponding to the key in a second array in memory, and each element in the second array is used to store the value;
[0120] An element position search module 440 is used to search the index file according to the target file position to obtain the element position of the value corresponding to the target key in the second array;
[0121] The value search module 450 is used to obtain the value corresponding to the target key from the second array according to the element position of the value corresponding to the target key in the second array.
[0122] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0123] Please refer to Figure 5 , Figure 5 A structural diagram of an electronic device for executing a data storage method or a data query method provided in an embodiment of the present application, the electronic device may include: at least one processor 510, such as a CPU, at least one communication interface 520, at least one memory 530 and at least one communication bus 540. Among them, the communication bus 540 is used to realize direct connection and communication between these components. Among them, the communication interface 520 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 530 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 530 can optionally also be at least one storage device located away from the aforementioned processor. Computer-readable instructions are stored in the memory 530. When the computer-readable instructions are executed by the processor 510, the electronic device executes the above-mentioned Figure 1 or Figure 2 The method process is shown.
[0124] Understandably, Figure 5 The structure shown is for illustration only, and the electronic device may also include Figure 5 More or fewer components as shown, or with Figure 5 Different configurations are shown. Figure 5Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0125] The present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the following is performed: Figure 1 or Figure 2 The method process in the method embodiment shown is executed by the electronic device.
[0126] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining data to be stored, wherein the data to be stored is key-value type data; calculating and obtaining a hash value to be stored corresponding to a key to be stored in the data to be stored; according to the hash value to be stored, storing the key to be stored at a corresponding position in an index file of a disk, and obtaining position information of the corresponding position; storing the hash value to be stored and the position information in a first array of a memory; storing the value corresponding to the key to be stored in a second array of the memory, and storing the element position where the value corresponding to the key to be stored in the second array is located at the position corresponding to the key to be stored in the index file.
[0127] In summary, the embodiments of the present application provide a data storage method, a data query method, a device and an electronic device. The data storage method maintains a first array that stores the position information of a hash value and a key in an index file, and maintains a second array that stores the value, and stores the position of the key and the value in the second array by constructing an index file. In this way, only the first array and the second array need to be maintained in the memory, and the index file is stored on the disk. Compared with the existing method of using a hash table to store data, this solution can effectively reduce the consumption of memory space.
[0128] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0129] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0130] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0131] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.
[0132] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A data storage method, It is characterized in that The method comprises: Acquire the data to be stored, where the data to be stored is of key-value type; Calculate and obtain the hash value to be stored corresponding to the key to be stored in the data to be stored; According to the hash value to be stored, the key to be stored is stored at a corresponding position in the index file of the disk, and the position information of the corresponding position is obtained; Storing the hash value to be stored and the location information in a first array in a memory; The value corresponding to the key to be stored is stored in a second array in the memory, and the element position where the value corresponding to the key to be stored is located in the second array is stored at the position corresponding to the key to be stored in the index file.
2. The method according to claim 1, It is characterized in that The step of storing the value corresponding to the key to be stored in a second array in the memory includes: Determine whether there is a value in the second array of the memory that is the same as the value corresponding to the key to be stored; If so, replace the same value with the value corresponding to the key to be stored and store it in the second array.
3. The method according to claim 1, It is characterized in that According to the hash value to be stored, storing the key to be stored at a corresponding position in the index file of the disk includes: According to the hash value to be stored, the key to be stored is stored in the row corresponding to the hash value to be stored in the index file of the disk; The obtaining the location information of the corresponding location includes: Gets the byte offset of the line.
4. The method according to claim 1, It is characterized in that The element position where the value corresponding to the key to be stored is located is the array element offset corresponding to the value, and the length of the array element offset is consistent with the maximum length of the elements in the second array.
5. The method according to claim 4, It is characterized in that The step of storing the element position of the value corresponding to the key to be stored in the second array at the position corresponding to the key to be stored in the index file includes: If the length of the array element offset corresponding to the value corresponding to the key to be stored is less than the maximum length, the array element offset is padded with zeros to make the length of the array element offset equal to the maximum length; The zero-filled array element offset is stored in the index file at a position corresponding to the key to be stored.
6. A data query method, It is characterized in that The method comprises: Obtain the target data to be queried, where the target data is key-value type data; Calculate and obtain the target hash value corresponding to the target key in the target data; Searching and obtaining the target file position corresponding to the target hash value from a first array in memory, wherein each element in the first array is used to store the hash value of the key and the corresponding file position, the file position is used to indicate the position of the key in the index file of the disk, the index file also stores the element position of the value corresponding to the key in a second array in memory, and each element in the second array is used to store the value; Search the index file according to the target file position to obtain the element position of the value corresponding to the target key in the second array; According to the element position of the value corresponding to the target key in the second array, the value corresponding to the target key is obtained from the second array.
7. A data storage device, It is characterized in that The device comprises: A data acquisition module is used to acquire data to be stored, wherein the data to be stored is key-value type data; A hash calculation module, used to calculate and obtain a hash value to be stored corresponding to a key to be stored in the data to be stored; A key storage module, used to store the key to be stored at a corresponding position in the index file of the disk according to the hash value to be stored, and obtain position information of the corresponding position; A hash value storage module, used for storing the hash value to be stored and the position information in a first array of a memory; The value storage module is used to store the value corresponding to the key to be stored in the second array of the memory, and store the element position of the value corresponding to the key to be stored in the second array at the position corresponding to the key to be stored in the index file.
8. A data query device, It is characterized in that The device comprises: A query data acquisition module is used to acquire target data to be queried, where the target data is key-value type data; A data hash calculation module, used to calculate and obtain a target hash value corresponding to a target key in the target data; A file location search module is used to search for a target file location corresponding to the target hash value from a first array in memory, wherein each element in the first array is used to store the hash value of the key and the corresponding file location, the file location is used to indicate the location of the key in the index file of the disk, the index file also stores the element location of the value corresponding to the key in a second array in memory, and each element in the second array is used to store the value; An element position search module, used to search the index file according to the target file position to obtain the element position of the value corresponding to the target key in the second array; The value search module is used to obtain the value corresponding to the target key from the second array according to the element position of the value corresponding to the target key in the second array.
9. An electronic device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is executed.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is executed.
