Data storage method and apparatus based on structure-optimized trie

Abstract

This invention discloses a data storage method and apparatus based on a structure-optimized trie. The method includes: identifying high-frequency characters in the data to be stored, where high-frequency characters are those whose frequency in the data exceeds a frequency domain threshold; counting the number of child nodes corresponding to the nodes of the high-frequency characters; determining the initial array length of the initial array required to store the data to be stored based on a first number of high-frequency characters, the number of child nodes, and a second number of ordinary characters in the data to be stored; storing all elements corresponding to the data to be stored in an array manner when the initial array length does not exceed the array length threshold; and storing all elements corresponding to the data to be stored in a hybrid storage manner when the initial array length exceeds the array length threshold. This invention solves the technical problem in related technologies where the trie structure used for text matching cannot achieve a good balance between space saving and performance, resulting in low reliability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data storage method and apparatus based on a structure-optimized trie. Background Technology

[0002] Trie trees are frequently used in text matching. They are used for searching Unicode text, where each node corresponds to a Unicode word, and each non-leaf node is associated with a set of child nodes. In data structure design, this set of child nodes can typically be a sorted array or a hash table. Using an array saves space, but its performance is poor. Using a hash table can improve processing speed, but it consumes a large amount of storage space.

[0003] There is currently no effective solution to the problem that the trie structure used in the aforementioned technologies for data storage cannot achieve a good balance between space saving and performance, resulting in low reliability. Summary of the Invention

[0004] This invention provides a data storage method and apparatus for a trie based on structure optimization, which at least solves the technical problem in related technologies where the trie structure used for text matching cannot achieve a good balance between space saving and performance, resulting in low reliability.

[0005] According to one aspect of the present invention, a data storage method based on a structure-optimized trie is provided, comprising: acquiring data to be stored, wherein the data to be stored is data for which storage operations need to be performed; determining high-frequency characters in the data to be stored, wherein the high-frequency characters are characters in the data to be stored whose frequency of occurrence is higher than a frequency domain threshold; counting the number of child nodes of the node corresponding to the high-frequency character; determining the initial array length of an initial array required to store the data to be stored based on a first number of high-frequency characters, the number of child nodes, and a second number of ordinary characters in the data to be stored, wherein the ordinary characters are characters in the data to be stored excluding the high-frequency characters; storing all elements corresponding to the data to be stored in an array manner when the initial array length is determined not to exceed an array length threshold; and storing all elements corresponding to the data to be stored in a hybrid storage manner when the initial array length is determined to exceed the array length threshold, wherein the hybrid storage manner includes a storage method comprising a hash table and an array.

[0006] Optionally, after determining the initial array length of the initial array required when storing the data to be stored using an array, the data storage method based on a structure-optimized trie further includes: determining the total number of high-frequency words in the data to be stored; and determining the array length threshold based on the total number.

[0007] Optionally, when the initial array length does not exceed the array length threshold, all elements corresponding to the data to be stored are stored in an array, including storing the ordinary characters, the high-frequency characters, and the child nodes in the array.

[0008] Optionally, when it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored using a hybrid storage method, including: determining the storage capacity of the hash table in the hybrid storage method based on the product of the array length threshold and a preset value, wherein the preset value is a value determined based on the key value of a common word and the array length threshold, and the common word is a word whose usage frequency is within a preset usage frequency range; storing the high-frequency words and their child nodes in the data to be stored according to the storage capacity of the hash table; and storing the data in the data to be stored excluding the data stored in the hash table using an array storage method.

[0009] Optionally, after determining the storage capacity of the hash table in the hybrid storage method based on the product of the array length threshold and the preset value, the data storage method based on the structure-optimized trie further includes: sorting all characters in the character set corresponding to the data to be stored in descending order according to their usage frequency to obtain the character set after descending order processing; and dividing the character set after descending order processing into a first sub-data set to be stored and a second sub-data set to be stored according to the storage capacity.

[0010] Optionally, when it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored in a hybrid storage method, including: storing the first data to be stored in the hash table in the hybrid storage method, and storing the second data to be stored in the array method.

[0011] According to another aspect of the present invention, a data storage device based on a structure-optimized trie is also provided, comprising: an acquisition unit for acquiring data to be stored, wherein the data to be stored is data for which storage operations need to be performed; a first determining unit for determining high-frequency words in the data to be stored, wherein the high-frequency words are words in the data to be stored whose frequency of occurrence is higher than a frequency domain threshold; a statistics unit for counting the number of child nodes of the node corresponding to the high-frequency words; a second determining unit for determining the initial array length of the initial array required to store the data to be stored based on a first number of high-frequency words, the number of child nodes, and a second number of ordinary words in the data to be stored, wherein the ordinary words are words in the data to be stored excluding the high-frequency words; a first storage unit for storing all elements corresponding to the data to be stored in an array manner when the initial array length is determined not to exceed the array length threshold; and a second storage unit for storing all elements corresponding to the data to be stored in a hybrid storage manner when the initial array length is determined to exceed the array length threshold, wherein the hybrid storage manner includes a storage method comprising a hash table and an array.

[0012] Optionally, the data storage device based on the structure-optimized trie further includes: a second determining unit, configured to determine the total number of high-frequency words in the data to be stored after determining the initial array length of the initial array required when the data to be stored is stored by an array; and a third determining unit, configured to determine the array length threshold based on the total number.

[0013] Optionally, the first storage unit includes: a first storage module, used to store the ordinary characters, the high-frequency characters, and the child nodes into an array.

[0014] Optionally, the second storage unit includes: a determining module, configured to determine the storage capacity of the hash table in the hybrid storage method based on the product of the array length threshold and a preset value, wherein the preset value is a value determined based on the key value of a common word and the array length threshold, and the common word is a word whose usage frequency is within a preset usage frequency range; a second storage module, configured to store the high-frequency words and the child nodes of the high-frequency words in the data to be stored according to the storage capacity of the hash table; and a third storage module, configured to store the data in the data to be stored excluding the data stored in the hash table using an array storage method.

[0015] Optionally, the data storage device based on the structure-optimized trie further includes: a sorting module, used to sort all the characters in the character set corresponding to the data to be stored in descending order according to their usage frequency after determining the storage capacity of the hash table in the hybrid storage method based on the product of the array length threshold and a preset value, to obtain the character set after descending order processing; and a partitioning module, used to divide the character set after descending order processing into a first sub-data set to be stored and a second sub-data set to be stored according to the storage capacity.

[0016] Optionally, the second storage unit includes: a fourth storage module, used to store the first data to be stored in the hash table in the hybrid storage method, and to store the second data to be stored in the array method.

[0017] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes the data storage method based on the structure-optimized trie described in any one of the foregoing embodiments.

[0018] According to another aspect of the present invention, a processor is also provided, the processor being configured to run a program, wherein the program, when running, executes the data storage method based on the structure-optimized trie described in any one of the foregoing embodiments.

[0019] In this embodiment of the invention, the following steps are taken: First, data to be stored is acquired, where the data to be stored is the data for which storage operations need to be performed. Second, high-frequency words in the data to be stored are determined, where high-frequency words are words in the data to be stored whose frequency is higher than a frequency domain threshold. Third, the number of child nodes corresponding to the nodes of the high-frequency words is counted. Fourth, the initial array length of the initial array required to store the data to be stored is determined based on the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored, where ordinary words are words in the data to be stored excluding high-frequency words. Fifth, when the initial array length does not exceed the array length threshold, all elements corresponding to the data to be stored are stored using an array. Sixth, when the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored using a hybrid storage method, where the hybrid storage method includes both hash tables and arrays. The above technical solution enables the determination of the storage method based on the data characteristics of the data to be stored before data storage, so that the determined storage method can effectively balance between search performance and storage space. This achieves the technical effect of improving data search performance and saving data storage space. It also solves the technical problem in related technologies where the trie structure used for text matching cannot achieve a good balance between saving space and performance, resulting in low reliability. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0021] Figure 1 This is a hardware structure block diagram of a mobile terminal for a data storage method based on a structure-optimized trie according to an embodiment of the present invention.

[0022] Figure 2 This is a flowchart of a data storage method based on a structure-optimized trie according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of a data storage device based on a structure-optimized trie according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] As described in the background section, in practical applications, the number of child nodes for a given node varies greatly. For example, a few high-frequency characters or affixes may have many child nodes, while most words have only a few. If arrays are used to store child nodes, the performance advantage of searching for high-frequency words cannot be realized. If hash tables are used to store child nodes, most common words and rare characters will occupy a lot of storage space. Existing data storage structures generally use only one storage method, failing to balance the two. To address these shortcomings, embodiments of this invention provide a data storage method and apparatus based on a structure-optimized trie.

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] The methods and embodiments provided in this invention can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal using a data storage method based on a structure-optimized trie, according to an embodiment of the present invention. For example... Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data storage method based on the structure-optimized trie in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] According to an embodiment of the present invention, a method embodiment of a data storage method based on a structure-optimized trie is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] Figure 2 This is a flowchart of a data storage method based on a structure-optimized trie according to an embodiment of the present invention, such as... Figure 2 As shown, the data storage method based on the structure-optimized trie includes the following steps:

[0032] Step S202: Obtain the data to be stored, wherein the data to be stored is the data for which a storage operation needs to be performed.

[0033] Optionally, the data to be stored may be data generated in actual business operations, such as data generated by various bank business processes or data generated by various departments in the academic affairs management system. No specific limitation is made here.

[0034] Step S204: Determine the high-frequency words in the data to be stored, wherein the high-frequency words are words in the data to be stored whose frequency is higher than the frequency domain threshold.

[0035] In this embodiment, the data to be stored can be analyzed to identify words that appear frequently and define them as high-frequency words.

[0036] Among them, the node corresponding to the high-frequency word usually has multiple child nodes.

[0037] Step S206: Count the number of child nodes corresponding to the high-frequency words.

[0038] In this embodiment, the number of child nodes corresponding to the high-frequency words can be counted, which facilitates the subsequent determination of the storage method for the data to be stored, making the determined data storage method easier to find the data.

[0039] Step S208: Determine the initial array length of the initial array required to store the data to be stored based on the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored, where ordinary words are words in the data to be stored excluding high-frequency words.

[0040] In this embodiment, the number of all high-frequency words can be determined, i.e., the first number; then, based on the first number, the number of child nodes determined above, and the number of each ordinary word in the data to be stored (i.e., the second number), the array length of the initial array required to store the data to be stored is determined, and the storage method of the subsequent data to be stored is determined.

[0041] Step S210: When it is determined that the initial array length does not exceed the array length threshold, all elements corresponding to the data to be stored are stored in an array manner.

[0042] In this embodiment, if it is determined that the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored do not exceed the array length threshold, the nodes corresponding to the high-frequency words, the child nodes, and the nodes corresponding to the ordinary words in the data to be stored can be stored together in the array to facilitate subsequent data retrieval.

[0043] Step S212: When it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored using a hybrid storage method, which includes both hash tables and arrays.

[0044] In this embodiment, when the initial array length exceeds the array length threshold, storing data using the array length will result in longer search times for high-frequency words. Therefore, a hybrid storage method is used to store the data to be stored, so that a balance can be found between search performance and storage space.

[0045] As described above, in this embodiment of the invention, after acquiring the data to be stored, high-frequency words in the data to be stored are determined, and the number of child nodes corresponding to the high-frequency words is counted. The initial array length of the initial array required to store the data to be stored is determined based on the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored. When the initial array length does not exceed the array length threshold, all elements corresponding to the data to be stored are stored in an array manner. When the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored in a hybrid storage manner. This achieves the goal of determining the storage method based on the data characteristics of the data to be stored before data storage, so that the determined storage method can effectively balance search performance and storage space, thereby improving the technical effect of data search performance and saving data storage space.

[0046] Therefore, the above-mentioned technical solution of the present invention solves the technical problem in the related art where the structure of the trie used for text matching cannot achieve a good balance between saving space and performance, resulting in low reliability.

[0047] According to the above embodiments of the present invention, after determining the initial array length of the initial array required when storing the data to be stored by an array, the data storage method based on the structure optimization trie further includes: determining the total number of high-frequency words in the data to be stored; and determining an array length threshold based on the total number.

[0048] In this embodiment, the total number of high-frequency words can be determined, and then the array length threshold can be determined based on the total number of high-frequency words.

[0049] Here, by considering the specific needs of a trie in actual business scenarios, a compromise structure is used to replace arrays or hash tables, achieving both space savings and good lookup speed. This data structure is essentially an array, but when the array size exceeds a threshold n (array length threshold), the internal storage elements are expanded into two parts: the first n*k elements are stored according to the key values, and the remaining elements are processed according to a sorted array.

[0050] For example, if the number of high-frequency words is 2000, the array length threshold can be set to 2000. In this way, the array length threshold can be dynamically adjusted.

[0051] According to the above embodiments of the present invention, when it is determined that the initial array length does not exceed the array length threshold, storing all elements corresponding to the data to be stored in an array manner may include: storing ordinary characters, high-frequency characters and child nodes into the array.

[0052] In this embodiment, when the initial array length does not exceed the array length threshold, all nodes can be stored in the value array in an array manner to improve data retrieval performance.

[0053] According to the above embodiments of the present invention, when it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored in a hybrid storage method, including: determining the storage capacity of the hash table in the hybrid storage method according to the product of the array length threshold and a preset value, wherein the preset value is a value determined according to the key value of a common word and the array length threshold, and common words are words whose usage frequency is within a preset usage frequency range; storing high-frequency words and their child nodes in the data to be stored according to the storage capacity of the hash table; and storing the data in the data to be stored excluding the data stored in the hash table in the array storage method.

[0054] In this embodiment, when it is determined that the initial array length does not exceed the array length threshold, the storage capacity of the hash table in the hybrid storage method can be determined first based on the product of the array length threshold and the preset value.

[0055] The preset values ​​here can be determined based on the key values ​​of ordinary characters and the array length threshold. The value of k must ensure that the key values ​​within (n*k) can include most commonly used characters. For example, when using Unicode directly as the key value of a character, the key values ​​of most characters are between 1 and 65536, so k = 65536 / n.

[0056] In reality, the number of Chinese character codes is much smaller than 65535. If using character index values ​​sorted by frequency of use, the value of n can be reduced to 200 and the value of k to 5000. This method can be used when memory is limited.

[0057] The values ​​of n and k mentioned above can be adjusted within a certain range, which allows the balance between data retrieval and storage space to be dynamically adjusted, thus improving the flexibility of data storage.

[0058] According to the above embodiments of the present invention, after determining the storage capacity of the hash table in the hybrid storage method based on the product of the array length threshold and the preset value, the data storage method based on the structure optimization trie further includes: sorting all characters in the character set corresponding to the data to be stored in descending order according to their usage frequency to obtain the character set after descending order processing; and dividing the character set after descending order processing into a first sub-data to be stored and a second sub-data to be stored according to the storage capacity.

[0059] In this embodiment, in order to improve the lookup speed of high-frequency words by storing them in a hash table, the data to be stored can be sorted in descending order according to the frequency of use. Then, the data to be stored can be split according to the length of (n*k) to obtain the data corresponding to (n*k) and the data after (n*k).

[0060] By using the above method, compared to the problem of Chinese character Unicode encoding values ​​being too scattered, the sorted Chinese character index values ​​will be used in the hash algorithm, increasing the probability and efficiency of using mixed storage in the child node array.

[0061] According to the above embodiments of the present invention, when it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored in a hybrid storage method, including: storing the first data to be stored in a hash table in the hybrid storage method, and storing the second data to be stored in an array method.

[0062] In this embodiment, a hash table can be used to store the first (n*k) elements after descending sorting, and an array can be used to store the remaining (n*k) elements.

[0063] The technical solution provided by the above embodiments of the present invention, combined with the characteristics of trie in actual business, uses a compromise structure to replace arrays or hash tables, achieving both space saving and good search speed. This data structure is essentially an array, but when the array size exceeds a threshold n, the internal storage elements are expanded into two parts: the first n*k elements are stored and searched according to their index values ​​(i.e., Unicode values ​​are hash values), and the elements after n*k are stored and searched according to a sorted array. Because the Unicode values ​​of most currently used characters are within the n*k range (i.e., 1-65536), most searches can be found using indexing, with only a few requiring binary search.

[0064] Based on the above method, high-frequency words will be stored using a hybrid hash structure, while ordinary words will still be stored using arrays. The memory usage of the entire trie structure can be balanced between pure arrays and pure hashes by adjusting the threshold n and the coefficient k, thereby optimizing the search performance accordingly.

[0065] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0067] According to embodiments of the present invention, a data storage apparatus for a structure-optimized trie is also provided for implementing the above-described data storage method for a trie based on structure optimization. Figure 3 This is a schematic diagram of a data storage device based on a structure-optimized trie according to an embodiment of the present invention, such as... Figure 3 As shown, the device includes: an acquisition unit 301, a first determination unit 303, a statistics unit 305, a second determination unit 307, a first storage unit 309, and a second storage unit 311. The data storage device based on a structure-optimized trie will be described in detail below.

[0068] The acquisition unit 301 is used to acquire data to be stored, wherein the data to be stored is the data for which a storage operation needs to be performed.

[0069] The first determining unit 303 is used to determine high-frequency words in the data to be stored, wherein the high-frequency words are words in the data to be stored whose frequency is higher than the frequency domain threshold.

[0070] The statistics unit 305 is used to count the number of child nodes corresponding to the node of a high-frequency word.

[0071] The second determining unit 307 is used to determine the initial array length of the initial array required to store the data to be stored based on the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored, wherein ordinary words are words in the data to be stored excluding high-frequency words.

[0072] The first storage unit 309 is used to store all elements corresponding to the data to be stored in an array manner when the initial array length is determined to be no more than the array length threshold.

[0073] The second storage unit 311 is used to store all elements corresponding to the data to be stored in a hybrid storage method when the initial array length exceeds the array length threshold. The hybrid storage method includes a storage method that combines a hash table and an array.

[0074] It should be noted that the above-mentioned acquisition unit 301, first determination unit 303, statistics unit 305, second determination unit 307, first storage unit 309 and second storage unit 311 correspond to steps S202 to S212 in the above embodiments. The six units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0075] As can be seen from the above, in the scheme described in the above embodiments of the present invention, the acquisition unit can acquire the data to be stored, wherein the data to be stored is the data for which storage operations need to be performed; the first determining unit determines the high-frequency words in the data to be stored, wherein the high-frequency words are words in the data to be stored whose frequency is higher than the frequency domain threshold; the statistics unit counts the number of child nodes of the node corresponding to the high-frequency words; the second determining unit determines the initial array length of the initial array required to store the data to be stored based on the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored, wherein ordinary words are words in the data to be stored excluding high-frequency words; when the first storage unit determines that the initial array length does not exceed the array length threshold, it stores all elements corresponding to the data to be stored in an array manner; when the second storage unit determines that the initial array length exceeds the array length threshold, it stores all elements corresponding to the data to be stored in a hybrid storage manner, wherein the hybrid storage manner includes a hash table and an array storage manner. This achieves the goal of determining the storage method based on the data characteristics of the data to be stored before data storage, so that the determined storage method can effectively balance the search performance and storage space, thereby improving the technical effect of data search performance and saving data storage space.

[0076] Therefore, the above-mentioned technical solution of the present invention solves the technical problem in the related art where the structure of the trie used for text matching cannot achieve a good balance between saving space and performance, resulting in low reliability.

[0077] In an optional embodiment, the data storage device based on the structure-optimized trie further includes: a second determining unit, configured to determine the total number of high-frequency words in the data to be stored after determining the initial array length of the initial array required when storing the data to be stored by array; and a third determining unit, configured to determine an array length threshold based on the total number.

[0078] In one alternative embodiment, the first storage unit includes: a first storage module for storing ordinary characters, high-frequency characters, and child nodes into an array.

[0079] In one optional embodiment, the second storage unit includes: a determining module, configured to determine the storage capacity of the hash table in the hybrid storage method based on the product of an array length threshold and a preset value, wherein the preset value is a value determined based on the key value of a common word and the array length threshold, and common words are words whose usage frequency is within a preset usage frequency range; a second storage module, configured to store high-frequency words and their child nodes in the data to be stored according to the storage capacity of the hash table; and a third storage module, configured to store the data to be stored excluding the data stored in the hash table in the data to be stored using an array storage method.

[0080] In an optional embodiment, the data storage device based on the structure-optimized trie further includes: a sorting module, used to sort all the characters in the character set corresponding to the data to be stored in descending order according to their usage frequency after determining the storage capacity of the hash table in the hybrid storage method based on the product of the array length threshold and a preset value, to obtain a character set after descending order processing; and a partitioning module, used to divide the character set after descending order processing into a first sub-data set to be stored and a second sub-data set to be stored according to the storage capacity.

[0081] In one optional embodiment, the second storage unit includes: a fourth storage module, used to store the first data to be stored in a hash table in a hybrid storage method, and to store the second data to be stored in an array method.

[0082] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes the data storage method based on a structure-optimized trie according to any one of the above.

[0083] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any communication device in a group of communication devices.

[0084] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring data to be stored, wherein the data to be stored is data for which storage operations need to be performed; determining high-frequency words in the data to be stored, wherein high-frequency words are words in the data to be stored whose frequency is higher than a frequency domain threshold; counting the number of child nodes corresponding to the nodes of the high-frequency words; determining the initial array length of the initial array required to store the data to be stored based on a first number of high-frequency words, the number of child nodes, and a second number of ordinary words in the data to be stored, wherein ordinary words are words in the data to be stored excluding high-frequency words; when the initial array length is determined not to exceed an array length threshold, storing all elements corresponding to the data to be stored in an array manner; when the initial array length is determined to exceed an array length threshold, storing all elements corresponding to the data to be stored in a hybrid storage manner, wherein the hybrid storage manner includes a storage method that combines a hash table and an array.

[0085] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: after determining the initial array length of the initial array required when storing the data to be stored by an array, determining the total number of high-frequency words in the data to be stored; and determining an array length threshold based on the total number.

[0086] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: storing ordinary characters, high-frequency characters, and child nodes into an array.

[0087] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining the storage capacity of the hash table in the hybrid storage method based on the product of an array length threshold and a preset value, wherein the preset value is a value determined based on the key value of a common word and the array length threshold, and common words are words whose usage frequency is within a preset usage frequency range; storing high-frequency words and their child nodes in the data to be stored according to the storage capacity of the hash table; and storing the data in the data to be stored excluding the data stored in the hash table using an array storage method.

[0088] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: after determining the storage capacity of the hash table in the hybrid storage method according to the product of the array length threshold and the preset value, sorting all the characters of the character set corresponding to the data to be stored in descending order according to the frequency of use to obtain the character set after descending order processing; dividing the character set after descending order processing into a first sub-data to be stored and a second sub-data to be stored according to the storage capacity.

[0089] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: storing the first data to be stored in a hash table in a hybrid storage method, and storing the second data to be stored in an array.

[0090] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes any of the above-described data storage methods based on a structure-optimized trie.

[0091] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0092] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0097] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data storage method based on a structure-optimized trie, characterized in that, include: Obtain the data to be stored, wherein the data to be stored is the data for which a storage operation needs to be performed; Identify high-frequency words in the data to be stored, wherein the high-frequency words are words in the data to be stored whose frequency of occurrence is higher than a frequency domain threshold; Count the number of child nodes corresponding to the nodes of the high-frequency words; The initial array length of the initial array required to store the data to be stored is determined based on the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored, wherein the ordinary words are the words in the data to be stored excluding the high-frequency words; When the initial array length is determined to be no more than the array length threshold, all elements corresponding to the data to be stored are stored in an array manner. When it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored using a hybrid storage method, wherein the hybrid storage method includes a storage method that combines hash tables and arrays.

2. The data storage method based on a structure-optimized trie according to claim 1, characterized in that, After determining the initial array length of the initial array required when storing the data to be stored, the method further includes: Determine the total number of high-frequency words in the data to be stored; The array length threshold is determined based on the total number.

3. The data storage method based on a structure-optimized trie according to claim 1, characterized in that, When the initial array length is determined to be no more than the array length threshold, all elements corresponding to the data to be stored are stored in an array format, including: Store the common characters, the high-frequency characters, and the child nodes into an array.

4. The data storage method based on a structure-optimized trie according to claim 1, characterized in that, When it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored using a hybrid storage method, including: The storage capacity of the hash table in the hybrid storage method is determined by multiplying the array length threshold and a preset value, wherein the preset value is a value determined by the key value of a common word and the array length threshold, and the common word is a word whose usage frequency is within a preset usage frequency range; Store the high-frequency words and their child nodes in the data to be stored according to the storage capacity of the hash table; The data to be stored, excluding the data stored in the hash table, is stored using an array storage method.

5. The data storage method based on a structure-optimized trie according to claim 4, characterized in that, After determining the storage capacity of the hash table in the hybrid storage method based on the product of the array length threshold and a preset value, the method further includes: Sort all characters in the character set corresponding to the data to be stored in descending order according to their frequency of use to obtain the character set after descending order processing; Based on the storage capacity, the character set after descending order processing is divided into a first sub-data to be stored and a second sub-data to be stored.

6. The data storage method based on a structure-optimized trie according to claim 5, characterized in that, When it is determined that the initial array length exceeds the array length threshold, all elements corresponding to the data to be stored are stored using a hybrid storage method, including: The first data to be stored is stored in the hash table of the hybrid storage method, and the second data to be stored is stored in the array method.

7. A data storage device based on a structure-optimized trie, characterized in that, include: An acquisition unit is used to acquire data to be stored, wherein the data to be stored is data for which a storage operation needs to be performed; The first determining unit is used to determine high-frequency words in the data to be stored, wherein the high-frequency words are words in the data to be stored whose frequency of occurrence is higher than a frequency domain threshold; The statistics unit is used to count the number of child nodes corresponding to the high-frequency word; The second determining unit is used to determine the initial array length of the initial array required to store the data to be stored based on the first number of high-frequency words, the number of child nodes, and the second number of ordinary words in the data to be stored, wherein the ordinary words are words in the data to be stored excluding the high-frequency words; The first storage unit is used to store all elements corresponding to the data to be stored in an array manner when the initial array length is determined to be no more than the array length threshold. The second storage unit is used to store all elements corresponding to the data to be stored in a hybrid storage method when it is determined that the initial array length exceeds the array length threshold. The hybrid storage method includes a storage method that combines a hash table and an array.

8. The data storage device based on a structure-optimized trie according to claim 7, characterized in that, Also includes: The second determining unit is used to determine the total number of high-frequency words in the data to be stored after determining the initial array length of the initial array required when the data to be stored is stored by an array; The third determining unit is used to determine the array length threshold based on the total number.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes the data storage method based on a structure-optimized trie as described in any one of claims 1 to 6.

10. A processor, characterized in that, The processor is used to run a program, wherein the program executes the data storage method based on a structure-optimized trie as described in any one of claims 1 to 6.

Images