Efficient storage and retrieval of text data

Abstract

A method and system for efficient storage of data entries containing text data is disclosed. The method may include: accessing a plurality of data entries in a data set; arranging the plurality of data entries in the data set in lexical order; placing a predetermined number of the plurality of data entries in each of a plurality of sub-blocks; performing data compression on the plurality of data entries in each of the plurality of sub-blocks to reduce redundancy in the plurality of data entries and create compressed data entries; placing one or more sub-blocks in each of a plurality of page blocks; and storing each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the data set.

Description

Technical Field

[0001] The present disclosure relates generally to efficient storage and retrieval of text data, and more particularly to an improved method and system for efficient storage and retrieval of data containing text in a cloud computing environment. Background Art

[0002] Because large amounts of data are stored in cloud computing systems, the amount of storage space available is often small. In addition, because of the large data sets in which the data is stored, searching, identifying, and retrieving desired data in the data sets stored in the cloud computing system is often time-consuming and resource-intensive. This causes certain tasks to become overly cumbersome. For example, when the amount of data in a data set is very large, performing analysis on the data set becomes very resource-intensive. This is particularly true for non-relational data structures, such as columnar data structures containing text.

[0003] Therefore, there is a need for an improved method and system for efficiently storing, searching, and retrieving text data in a cloud computing environment. Summary of the invention

[0004] In one general aspect, the present application describes a device having a processor and a memory in communication with the processor, wherein the memory includes executable instructions that, when executed by the processor, cause the device to perform a plurality of functions. The functions may include: accessing a plurality of data entries in a data set; arranging the plurality of data entries in the data set in lexical order; placing a predetermined number of the plurality of data entries in each of a plurality of sub-blocks; performing data compression on the plurality of data entries in each of the plurality of sub-blocks to reduce redundancy in the plurality of data entries and create compressed data entries; placing one or more sub-blocks in each of a plurality of page blocks; and storing each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the data set.

[0005] In yet another general aspect, the present application describes a method for efficient storage of data entries containing text data. In one implementation, the method may include: accessing a plurality of data entries in a data set; arranging the plurality of data entries in the data set in lexical order; placing a predetermined number of the plurality of data entries in each of a plurality of sub-blocks; performing data compression on the plurality of data entries in each of the plurality of sub-blocks to reduce redundancy in the plurality of data entries and create compressed data entries; placing one or more sub-blocks in each of a plurality of page blocks; and storing each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the data set.

[0006] In another general aspect, the present application describes a non-transitory computer-readable medium having instructions stored thereon, which, when executed, cause a programmable device to: access multiple data entries in a data set; arrange the multiple data entries in the data set in lexical order; place a predetermined number of multiple data entries in each of a plurality of sub-blocks; perform data compression on the multiple data entries in each of the plurality of sub-blocks to reduce redundancy in the multiple data entries and create compressed data entries; and place one or more sub-blocks in each of a plurality of page blocks; and store each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the data set.

[0007] This summary is provided to introduce a selection of concepts that are further described in the detailed description below in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings depict one or more implementations according to the present teachings by way of example only and not limitation. In the drawings, like reference numerals refer to the same or similar elements. In addition, it should be understood that the accompanying drawings are not necessarily drawn to scale.

[0009] Figure 1 An exemplary system is depicted upon which aspects of the present disclosure may be implemented.

[0010] Figure 2 Depicted are exemplary data structure blocks for storing text data in a dataset.

[0011] Figure 3A-3B Depicted is an example of organizing data structure blocks in a host file system for storing data.

[0012] Figure 4 An exemplary structure for creating a data entry ID is depicted.

[0013] Figure 5 Depicted are examples of array data structures for storing data entries in a secondary data structure.

[0014] Figure 6 is a flowchart for efficiently storing data containing natural language text.

[0015] Figure 7is a flow chart depicting an exemplary method for efficiently compressing data entries for data containing natural language text.

[0016] Figure 8 is a block diagram illustrating an exemplary software architecture, portions of which may be used in conjunction with the various hardware architectures described herein.

[0017] Fig. 9 is a block diagram illustrating components of an exemplary machine configured to read instructions from a machine-readable medium and perform any of the features described herein. DETAILED DESCRIPTION

[0018] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. By reading this specification, those skilled in the art will appreciate that various aspects can be practiced without such details. In other cases, well-known methods, procedures, components and / or circuits have been described at a relatively high level without details in order to avoid unnecessarily obscuring various aspects of the present teachings.

[0019] In today's computing environment, it is often necessary to store large amounts of data in cloud computing systems. Examples of such data include text data, such as strings that make up dictionaries of different world languages ​​or data written as plain text strings in natural languages. This data is then frequently searched, for example, by performing a full-text search on all or part of the stored data. It is often important for users to be able to search quickly and efficiently. Currently available processes for data compression that operate uniformly on all types of stored data may not be applicable to text data. In addition, due to the unique properties of text strings, this type of data may be more suitable for alternative methods of compression and decompression.

[0020] To solve these technical problems and more, in an example, the present specification provides a technical solution for an efficient method of storing and searching data entries containing text data (e.g., data strings). In order to improve the current method of data storage, the technical solution uses two independent data structures: a primary file system and a secondary data structure. The primary file system can provide an efficient compression technology, wherein the data strings are sorted and stored in lexical order. Sorting the data entries in lexical order enables the system to use compression (e.g., delta compression) technology to eliminate overlapping characters. Then, the data entries can be divided into a plurality of sub-blocks, wherein each sub-block contains a predetermined number of data entries. The sub-blocks can be in lexical order, and each set of sub-blocks can form a physical page block. Because the sub-blocks are in lexical order, the starting characters at the beginning and end of the sub-blocks can be easily decrypted. This results in providing a very efficient mechanism for performing searches. Therefore, the technical solution not only provides a very efficient mechanism for compressing and storing data entries containing text data, but also provides a fast and efficient solution for searching the stored data.

[0021] The secondary data structure provided by the technical solution can be used to maintain and allow incoming mutations, updates, or create requests. This can prevent the main file system and the storage device on which the main file system is stored from being affected by memory usage efficiency, such as unnecessary locking, mutations, and changes. Instead, updates can be moved to one or more overflow pages. This process may be particularly effective for storing text data such as strings in a dictionary, because once the original dictionary is stored, new updates and additions are rare. Therefore, the technical solution provides a time and resource efficient mechanism for storing and searching a large number of data entries containing text data (e.g., strings).

[0022] Those skilled in the art will appreciate after reading this disclosure that the benefits and advantages provided by such technical solutions can include, but are not limited to: solutions to the technical problems of inefficient, resource-intensive storage and searching of large data sets containing text data in a computer environment. The technical solutions and implementations provided herein optimize and improve the process of compressing and storing data. Specifically, the technical solutions enable more efficient storage of text data and provide faster and more efficient searches for stored data, resulting in greatly improved storage systems and processes overall. The benefits provided by these technical solutions include providing increased efficiency, increased availability, and faster search results of the system by saving resources.

[0023] Figure 1An exemplary system 100 on which various aspects of the present disclosure may be implemented is illustrated. The system 100 may include a server 110 that may be connected to or include a data store 116 that may store data. The system 100 may also include client computer devices 112 and 114 that are connected to the server 110. The client computers 112 and 114 may be used to search or modify the data in the data store. The data store 116 and the client computer devices 112 and 114 may be connected via a network 130 or locally connected to the server 110 via one or more separate network connections. It should be noted that although only one server and data store are shown for simplicity, the system 100 may include multiple servers and data stores across which data is stored. In addition, the client computers 112 and 114 are merely exemplary client devices that are connected to or can access data in the data store 116.

[0024] Client devices 112 and 114 may be personal or handheld computing devices having both input elements and output elements. For example, client devices 112 and 114 may be one of the following: a mobile phone; a smart phone; a tablet computer; a phablet; a smartwatch; a wearable computer; a personal computer; a desktop computer; a laptop computer; a gaming device / computer; a television; etc. This list is for example purposes only and should not be considered limiting. In one implementation, client devices 112 and 114 may be servers. Figure 8 and Fig. 9 The internal hardware structure of the client device is discussed in more detail.

[0025] The network 110 may be a wired or wireless network or a combination of wired and wireless networks that connects one or more elements of the system 100. Users may be able to access and / or make changes to the data store 116 via client devices 112 and 114. For example, a user may access the data store 116, search for elements within the data store 116, and / or modify one or more data entries of the data store.

[0026] The data store 216 may include multiple data sets that can be searched, accessed, or modified. The data sets may include: relational data sets (e.g., structured query language (SQL) data sets), such as each table has a large number of rows; or non-relational data sets (NoSQL), such as columnar data sets for storing text data. Therefore, the system 100 can be used to support large database storage and retrieval systems, such as in a cloud computing environment. Client devices 114 and 112 can access data within the data sets via one or more applications. The application can be any user application that provides access to the underlying data.

[0027] Figure 2 An exemplary data structure block for storing text data in a data set is depicted. In one implementation, the primary file system for storing text data is divided into a plurality of sub-blocks, each of the plurality of sub-blocks containing a predetermined number of text data strings. In this manner, each of the sub-blocks can contain the same number of data strings. In an example, the number of data strings in each sub-block is 32. Thus, Figure 2 The file system block 200 may include 32 context data strings. Any other number of data strings may be selected for use in constructing the block.

[0028] Because the text data string in the data set is different in length usually, it will be advantageous to provide fast search capability in a mode that fully utilizes space to compress the main data string. In an example, the technical solution realizes this by first organizing the data string in each sub-block in lexical order. As described in file system 200, the data string in each sub-block can be organized in lexical order so that the data string is arranged in alphabetical order based on the alphabetical order of its composition letters. This enables the file system to eliminate the repeated characters between continuous strings.

[0029] In one implementation, elimination of repeated characters is achieved by comparing each data string to a previous data string sorted in lexical order, and identifying any starting characters that are identical to the starting characters of the previous data string. For example, a file system block 200 contains the data strings apple, appreciate, approach, ... cat. When comparing the data string "appreciate" to the previous data string "apple", it can be determined that both strings include the starting characters "app". As a result, instead of repeating those characters, the data string for "appreciate" can be compressed to eliminate them. This can be achieved by starting each string in the data set with the number of repeated characters in the string, followed by the number of non-repeated characters, and finally a list of non-repeated characters. For example, a compressed data string for "appreciate" can begin with the number 3 to indicate that the first three characters of the string are identical to the first three characters of the previous string. This number is followed by the number 7, which is the number of remaining characters in the data string, followed by the remaining characters. Thus, instead of storing the string "appreciate", the file system is able to store the string "37reciate". Similarly, the data string for approach may be converted to "44oach", indicating that the first four characters are the same as in the previous string "appreciate", and the remaining four characters are "aoch". If there are no repeated characters in the data string, the first number may be zero, followed by the total number of characters in the string. This is illustrated in string 32 of file system block 200, where the data string "cat" is converted to "03cat". Since the first data string in a block is not compared to a previous data string, the first data string may include only the number of characters in the data string, as illustrated by the compressed data entry for "apple" being converted to "5apple".

[0030] In this way, the technical solution can provide a cost and space efficient system to store text data by converting each data entry in a data structure sub-block into a compressed data entry, which starts with the length of the prefix common to the data entry and the previous data string in the block (if the data string is the first in the block, this will be the length of the data string). The next character in the compressed data entry can be the number of remaining characters in the data string after the common prefix. This number can be added unless the data string is the first in the sub-block. After ignoring the prefix characters of the previous data string, the number of remaining characters can be followed by the actual remaining characters of the data string. Since a large number of text data strings in lexical order include repeated starting characters, this scheme provides an opportunity to significantly reduce the number of characters stored in the data structure. This may be particularly useful when storing words in a dictionary, because when in lexical order, a word is a data string that usually starts with the same character as the previous word. Therefore, the technical solution significantly reduces the amount of memory required for storing text data containing a string. It should be noted that although the compression techniques discussed above relate to text data (e.g., strings), other types of compression techniques can be used to compress different types of data. For example, natural sorting mechanisms can be applied to different types of data.These compression techniques can also be used in combination with the data structures and search mechanisms discussed herein to improve the efficiency of storage space and resource access, as discussed further below.

[0031] Figure 3A-3B Depicts an example of organizing data structure sub-blocks in a main file system for storing data. Figure 3A As depicted in , the main file system 300 may include a plurality of sub-blocks (n blocks) organized in lexical order. The total number of sub-blocks in a main file system such as the main file system 300 may depend on the amount of data in the data set, and may change as the amount of data changes (e.g., data is deleted or added to the data set). As discussed above, each of the sub-blocks in the main file system may contain the same number of data entries. As a result, the sizes of the sub-blocks may differ from one another, depending on the number of characters and the amount of compression in the data string of each block. Therefore, different numbers of sub-blocks may be grouped together to form a physical page block.

[0032] Figure 3BAn example of assembling sub-blocks of the primary file system into physical page blocks is depicted. As depicted, each of the physical page blocks may contain a different number of sub-blocks. In one implementation, each physical page block corresponds to a page in a memory. As is known in the art, a memory page may be a fixed-length contiguous block of memory described by a single entry in a memory page table. As a result, each page may be accessed from the memory in a single page read. Because each physical page block may be a fixed-length block of memory (e.g., physical page blocks have the same size) and because the sub-blocks are of different sizes, each physical page block may contain a different number of sub-blocks. This may also result in a particular number of physical page blocks including some empty space. For example, Figure 3B Page block 1 of t includes blank space 310, while page block m includes blank space 320. As depicted, the size of blank spaces 310 and 320 may vary and may depend on the size of the sub-blocks in each physical page block.

[0033] In one implementation, the blank spaces 310 and 320 are reserved for future modifications to the sub-blocks. For example, if future modifications include adding one or more data entries to a sub-block and the size of the new data entries is larger than the data entries previously stored in the sub-block, the extra space can be used to modify the sub-block without rearranging how the sub-blocks are organized in the physical page blocks. In one implementation, overflow buckets and blocks are implemented in the blank spaces to prevent the need to rearrange strings of sub-blocks in the event of changes or additions. In an example, if the blank space overflows, one or more overflow pages are utilized to accommodate overflows from multiple different physical pages.

[0034] In another example, the blank space can be used to store metadata, such as index cache, hit counts, etc. If the blank space needs to be used as overflow space in the future, the data stored in the blank space can be transferred to a different location in the memory. In this way, the technical solution can ensure efficient use of the memory space.

[0035] Since the sub-blocks in each physical page block are in lexical order, the physical page blocks themselves are also in lexical order. Therefore, the first data string of each consecutive physical page block can indicate the content of the physical page block. For example, when the first data string stored in page block 1 is the word "apple", and the first data string stored in page block 2 is the word "candle", this shows that all data strings starting with letters between the words "apple" and "candle" are located in page block 1. This results in a very efficient mechanism for searching in the main file system.

[0036] In one implementation, a search in a column for a given value can be initiated by a client giving an ID for the value to be searched. To search for the value in the primary file system, a binary search can be performed on the ID at the physical page block level. Once the correct page block is located, a similar binary search can be performed to determine which sub-blocks contain the value. Similarly, because the sub-blocks are in lexical order, it will be easy to determine in which sub-block the value may be located by examining only the first data string in each sub-block. Once the correct sub-block is identified, it can be scanned sequentially to locate the searched value. Since each sub-block contains only a limited number of data entries (e.g., 32 data strings), the process of sequentially scanning sub-blocks is still fast and efficient.

[0037] In one implementation, to ensure efficient searching, an appropriate mechanism for identifying data entries may include assigning an ID to each data entry. Figure 4 Depicted is an exemplary structure for creating a data entry ID. In order to ensure that the ID properly and quickly identifies the location of the data entry, a portion 410 for identifying a physical block, a portion 420 for identifying a sub-block, and a portion 430 for identifying the data entry in the sub-block may be included. Therefore, the ID can be calculated in three parts, wherein the first part holds the ID of the physical block, the second part holds the ID of the sub-block, and the third part holds the ID of the data entry string within the sub-block. This may involve, for example, allocating a smaller number of bits to the position in the sub-block, allocating a larger number of bits to the position of the sub-block, and allocating a larger number of bits to the position of the physical block. In the example, 5 bits can be used for the position of the data entry in the sub-block (for example, when 32 data entries are allocated for each sub-block), 7 bits can be used for the sub-block block number, and 20 bits can be used for the physical block ID. Utilizing these numbers can allow the data structure to store more than 4 million data entries. By using an ID that includes an identification of each sub-structure in the sub-structure that organizes the data, the solution can ensure easy and efficient search.

[0038] In addition to the three-part ID that enables efficient searching, the ID can also easily enable the system to determine whether the searched value is in the primary file system or in the secondary file system. That is because the three-part ID is an unsigned integer of a predetermined number of bits, where each part is a bit representing a position. When read as an integer, the ID will have an integer value. Because the values ​​are continuous, the integer values ​​may be continuous. Therefore, the ID for the data entry stored in the primary file system will represent a series of integer values. Because it is easy to determine the integer values ​​for the first and last data entries in the primary file system, it is easy to confirm the range of the ID representing the data entry in the primary file system. Anything that falls outside this range can be identified as not being located in the primary file system. Therefore, a quick check of the ID value for the search term can determine whether the term is located in the primary file system or the secondary file system. This can provide a simple and efficient first-level search to locate the search term.

[0039] One of the ways in which the technical solution ensures efficiency is by providing a secondary data structure for managing modification, update, addition and / or creation requests so that the primary file system and its corresponding memory are not subject to unnecessary locking or changes. In this way, data modifications for the primary file system can be stored in the secondary data structure and implemented in batches. When the system determines that changes need to be made, the changes can be implemented at infrequent time intervals. For example, when the secondary data structure reaches a certain capacity level, indicating that it may soon run out of space, the changes stored in the secondary data structure can be transferred to the primary file system. Once the transfer is completed, the data entries in the secondary data structure can be removed to consider future entries. In one implementation, the merge between the data in the secondary data structure and the data in the primary file system can be performed based on a predetermined time interval. In an example, in order to avoid changing the ID when a merge occurs, the update from the secondary data structure is moved to the overflow page, and the ID for the updated data entry is generated accordingly. In this way, as long as a number of overflow pages are maintained, the entire system can operate efficiently.

[0040] In one implementation, the primary file system and the secondary data structure operate in a manner similar to a two-level log structured merge tree (LSM tree). As is known in the art, a two-level LSM tree may include two tree structures, one of which is smaller and resides entirely in memory, while the other is stored on disk. New records can be inserted into the memory-resident component until the insertion causes the component to exceed a certain size threshold. Once this limit is reached, consecutive entry segments can be removed from the smaller memory-resident component and merged into the larger component on disk. The performance characteristics of the LSM tree arise from the fact that each component is tuned according to the characteristics of its underlying storage medium, and data is efficiently migrated across the medium in rolling batches using an algorithm reminiscent of merge sort.

[0041] The primary and secondary data structures utilized by the present technical solution are similar to LSM trees in that they include two data structure components, one of which is smaller than the other, and changes are stored in the smaller data structure component (the secondary data structure) until they are transferred to the larger data structure in one or more batches. However, instead of a smaller data structure in memory and a larger data structure on disk, the structure utilized in the technical solution can be adapted to optimize the in-memory structure for both the primary and secondary data structures, or the case of an in-memory secondary data structure and a disk-based primary file system.

[0042] Because the secondary data structure is smaller in size than the primary file system and contains a limited number of data entries, it may not need to be compressed and optimized in the same manner as the primary file system. In fact, due to the relatively small number of entries in the secondary data structure, the probability of having repeated characters in a continuous string of data is not very high. Therefore, the use of the compression techniques discussed above may not result in a significant reduction in memory size. Therefore, in one implementation, the secondary data structure simply utilizes an array-like data structure in which entries are appended to the array as they enter.

[0043] Figure 5Depicted is an exemplary array data structure 500 for storing data entries in the secondary data structure. The array structure 500 of the secondary data structure can simply include a one-dimensional array for storing data entries. In one implementation, each entry in the array has an associated ID. Then, each associated ID is mapped to a value ID by a search tree to improve efficiency. In one implementation, a cache-sensitive data structure in memory can be used for the secondary data structure to achieve improved efficiency and high performance. Examples of cache types that can be used include a HashSet with amortized O(1) complexity or a binary tree cache with, for example, O(log n) complexity. This may involve maintaining multiple caches and range structures to perform fast and direct ID to data entry searches. The data structure can be a bidirectional cache structure that can provide searches from ID to data entry and from data entry to ID.

[0044] Figure 6 600 for efficiently storing text data. At 605, method 600 can begin by receiving a request to store a data set. For example, this can occur when a data set is first presented for storage in a new storage device. Alternatively, the process can be initiated to convert an existing data set into an improved data structure of the technical solution discussed in this article. In an example, the process can be automatically initiated as part of a reconstruction of a data storage device.

[0045] Once the request to store the data set has been received and processed, method 600 can sort the entries in the data set in lexical order at 610. In this way, all available data entries containing text data can be sorted in lexical order for faster processing and compression. Thereafter, the data entries have been sorted in lexical order, and at 615, they can be divided into sets of equal numbers of data entries to place each set in a sub-block of the data structure. Therefore, a predetermined number of data entries is placed in each sub-block. The predetermined number can be selected based on optimization factors, which can involve collecting experience and / or experimental evidence before deciding what the optimal predetermined number is. In the example, the predetermined number is 32, resulting in each sub-block having 32 data entries. It should be noted that the data entries maintain their lexical order when they are divided into sub-blocks. Therefore, once created, the resulting sub-blocks are automatically in lexical order.

[0046] After the data entries have been assigned to their respective sub-blocks, at 620, a compression technique, such as the compression process of method 700 discussed below, can be applied to the data entries to compress the entries in each sub-block. Once the data entries are placed in their respective sub-blocks and compressed, at 625, the sub-blocks can be constructed into physical page blocks that maintain lexical order. This can be accomplished by dividing the sub-blocks into multiple physical page blocks based on the size of the sub-blocks and the size of each physical page block. As discussed above, the physical page blocks have the same size. As a result, the sub-blocks can be assigned to corresponding physical page blocks based on the size of each sub-block and the remaining space available in the physical page blocks. This may result in different physical page blocks having different numbers of sub-blocks and having some additional empty space.

[0047] In addition to dividing the data entries into sub-blocks and physical blocks, the method 600 may also include assigning an ID to each data entry at 630 to ensure efficient search and retrieval of the data. The ID may be assigned by considering the physical block ID, the sub-block ID, and the location of the data entry within the sub-block. Once the ID has been assigned to all data entries, at 635, the method 600 may wait to receive a request to modify the data entry or add a data entry to the data set. Since the request to modify or add a data entry may require significant changes to the data structure, the request to modify or add a data entry may be processed by creating an entry in the secondary data structure at 640. This prevents the primary file system from being negatively affected each time a change is required to the underlying data. Instead, the changes may be stored in the secondary data structure until the method 600 determines at 645 that a condition has been made for merging the secondary data structure with the primary file system. In an example, the condition may be a determination that the secondary data structure has reached a predetermined capacity or that a predetermined time interval has passed since a previous merge.

[0048] When it is determined at 645 that the condition has been met, method 600 may continue at 650 to transfer the contents of the secondary data structure to the primary file system. This may involve the steps of organizing the data entries in lexical order in the secondary data structure, examining the physical block and then the sub-block to determine where in the data structure the new data entry should be located to maintain the lexical order, compressing the data entry before inserting it in the sub-block, and reorganizing the remaining entries to maintain the order. Alternatively, the additional data entries may be inserted into an overflow bucket of the empty space of the physical page. If it is determined at 645 that the condition has not been met, method 600 may return to step 635 to await receipt of the next request to modify or add a data entry.

[0049] Figure 7700 is a flowchart depicting an exemplary method 700 for efficiently compressing data entries for text data. At 705, method 700 may begin by checking a data entry to determine whether the data entry is the first data entry in a sub-block, at 710. If it is determined that the data entry is the first sub-block in the sub-block, then at 715, method 700 may continue to assign the number of characters in the data entry (e.g., the number of characters in the string) as a prefix for the compressed data entry. However, when it is determined that the data entry is not the first sub-block in the sub-block, at 720, method 700 may continue to compare the current data entry with the previous data entry in the sub-block to determine whether there is any overlap in its starting character at 725. If method 700 identifies any overlapping characters (e.g., the data entries start with the same character), method 700 may continue to assign the number of overlapping characters as a prefix, at 735. However, when no overlapping characters are identified at 725, method 700 may continue to assign a prefix of zero to the compressed data entry at 730.

[0050] Once the prefix has been assigned, the method 700 may proceed to assign the number of characters remaining in the received data entry to the compressed data entry at 740. Depending on whether there are any overlapping characters, this number may be the same as the number of characters in the data entry or equal to the total number of characters in the data entry minus the number of overlapping characters. After the prefix and the number of remaining characters (for any entry other than the first data entry) have been assigned, the method 700 may proceed to list the remaining characters in the data entry at 745. This results in a new compressed data entry consisting of a list of numbers and non-repeating characters.

[0051] Once the current data entry has been processed in this manner, method 700 may determine at 750 whether there are any other uncompressed data entries in the current sub-block. When it is determined that there are more data entries, method 700 may return to step 720 to compare the characters in the current data entry with the characters in the previous data entry and continue with the previously discussed steps as described above. However, if it is determined that there are no more data entries in the current sub-block, method 700 may continue to the next sub-block at 755 and then check the data entries in the next sub-block at 705. The steps of method 700 may be repeated until the end of the sub-block is reached.

[0052] Therefore, in various implementations, a technical solution is provided for an improved method and system for storing data entries in a data structure. The technical solution provides a compression mechanism for compressing data entries containing text data (e.g., data strings constituting a dictionary), and provides a mechanism for easy and efficient searching of the resulting data structure. In one implementation, the data structure is divided into a primary file system and a secondary data structure to prevent inefficient use of memory. Therefore, the technical solution provides an efficient mechanism for storing, searching, and accessing data in a data structure for data entries containing text data.

[0053] Figure 8 is a block diagram 800 illustrating an exemplary software architecture 802, portions of which may be used in conjunction with the various hardware architectures described herein, which may implement any of the features described above. Figure 8 808. Executable instructions 808 represent executable instructions of software architecture 802, including implementations of methods, modules, etc. described herein.

[0054] The hardware layer 804 also includes a memory / storage 810, which also includes executable instructions 808 and accompanying data. The hardware layer 804 may also include other hardware modules 812. The instructions 808 held by the processing unit 808 may be part of the instructions 808 held by the memory / storage 810.

[0055] Exemplary software architecture 802 can be conceptualized as layers, each providing various functions. For example, software architecture 802 may include layers and components, such as operating system (OS) 814, library 816, framework 818, application 820, and presentation layer 824. In operation, application 820 and / or other components within a layer may call API calls 824 to other layers and receive corresponding results 826. The illustrated layers are representative in nature, and other software architectures may include additional or different layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 818.

[0056] OS 814 can manage hardware resources and provide public services. OS 814 may include, for example, kernel 828, service 830 and driver 832. Kernel 828 may serve as an abstraction layer between hardware layer 804 and other software layers. For example, kernel 828 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Service 830 may provide other public services for other software layers. Driver 832 may be responsible for controlling or interfacing with underlying hardware layer 804. For example, driver 832 may include display driver, camera driver, memory / storage device driver, peripheral device driver (e.g., via universal serial bus (USB)), network and / or wireless communication driver, audio driver, etc., depending on hardware and / or software configuration.

[0057] The library 816 may provide a common infrastructure that may be used by the application 820 and / or other components and / or layers. The library 816 generally provides functions for use by other software modules to perform tasks, rather than directly interacting with the OS 814. The library 816 may include a system library 834 (e.g., a C standard library) that may provide functions such as memory allocation, string manipulation, file operations, etc. In addition, the library 816 may include an API library 836, such as a media library (e.g., supporting the presentation and manipulation of image, sound, and / or video data formats), a graphics library (e.g., an OpenGL library for rendering 2D and 3D graphics on a display), a database (e.g., SQLite or other relational database functions), and a network library (e.g., WebKit that may provide network browsing functions). The library 816 may also include a variety of other libraries 838 to provide many functions for the application 820 and other software modules.

[0058] Framework 818 (sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by applications 820 and / or other software modules. For example, framework 818 can provide various graphical user interface (GUI) functions, advanced resource management, or advanced positioning services. Framework 818 can provide a wide range of other APIs for applications 820 and / or other software modules.

[0059] Applications 820 include built-in applications 820 and / or third-party applications 822. Examples of built-in applications 820 may include, but are not limited to, contact applications, browser applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 822 may include any application developed by an entity other than the supplier of a particular system. Applications 820 may use the functionality available via OS 814, library 816, framework 818, and presentation layer 824 to create a user interface to interact with a user.

[0060] Some software architectures use virtual machines, as illustrated by virtual machine 828. Virtual machine 828 provides an execution environment in which applications / modules can run just as they would on a hardware machine (e.g., such as a Fig. 9 The virtual machine 828 may be hosted by a host OS (e.g., OS 814) or a hypervisor and may have a virtual machine monitor 826 that manages the operation of the virtual machine 828 and interoperates with the host operating system. A software architecture different from the software architecture 802 outside the virtual machine, such as an OS 850, a library 852, a framework 854, an application 856, and / or a presentation layer 858, may be run within the virtual machine 828.

[0061] Fig. 9 900 is a block diagram illustrating components of an exemplary machine 900, which is configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any of the features described herein. The exemplary machine 900 is in the form of a computer system, within which instructions 916 (e.g., in the form of software components) for causing the machine 900 to perform any of the features described herein can be executed. In this way, the instructions 916 can be used to implement the methods or components described herein. The instructions 916 cause the unprogrammed and / or unconfigured machine 900 to operate as a specific machine configured to perform the described features. The machine 900 can be configured to operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 900 can operate in the capacity of a server machine or a client machine in a server-client network environment, or operate as a node in a peer-to-peer or distributed network environment. The machine 900 may be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and / or entertainment system, a smartphone, a mobile device, a wearable device (e.g., a smart watch), and an Internet of Things (IoT) device. In addition, although only a single machine 900 is shown, the term "machine" includes a collection of machines that individually or jointly execute instructions 916.

[0062] The machine 900 may include a processor 910, a memory 930, and I / O components 950, which may be communicatively coupled via, for example, a bus 902. The bus 902 may include multiple buses that couple various elements of the machine 900 via various bus technologies and protocols. In an example, the processor 910 (including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 912a to 912n, which may execute instructions 916 and process data. In some examples, one or more processors 910 may execute instructions provided or identified by one or more other processors 910. The term "processor" includes a multi-core processor, which includes cores that can execute instructions simultaneously. Although Fig. 9 Multiple processors are shown, but the machine 900 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors each with a single core, multiple processors each with multiple cores, or any combination thereof. In some examples, the machine 900 may include multiple processors distributed across multiple machines.

[0063] The memory / storage 930 may include a main memory 932, a static memory 934 or other memory, and a storage unit 936, both of which can be accessed by the processor 910, such as via the bus 902. The storage unit 936 and the memory 932, 934 store instructions 916 that embody any one or more of the functions described herein. The memory / storage 930 may also store temporary, intermediate and / or long-term data for the processor 910. The instructions 916 may also reside completely or partially in the memory 932, 934 during execution thereof, in the storage unit 936, in at least one of the processors 910 (e.g., in a command buffer or cache memory), in a memory at at least one of the I / O components 950, or any suitable combination thereof. Therefore, the memory 932, 934, the storage unit 936, the memory in the processor 910, and the memory in the I / O component 950 are examples of machine-readable media.

[0064] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions and data that cause the machine 900 to operate in a specific manner. As used herein, the term "machine-readable medium" does not cover transient electrical or electromagnetic signals themselves (such as on a carrier wave propagating through the medium); therefore, the term "machine-readable medium" can be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible machine-readable media may include, but are not limited to: non-volatile memory (such as flash memory or read-only memory (ROM)), volatile memory (such as static random access memory (RAM) or dynamic RAM), cache memory, cache memory, optical storage media, magnetic storage media and devices, network accessible or cloud storage devices, other types of storage devices, and / or any suitable combination thereof. The term "machine-readable medium" applies to a single medium or a combination of multiple media used to store instructions (e.g., instructions 916) executed by the machine 900, such that the instructions, when executed by one or more processors 910 of the machine 900, cause the machine 900 to perform one or more features described herein. Thus, a “machine-readable medium” may refer to a single storage device, as well as a “cloud-based” storage system or storage network that includes multiple storage devices or devices.

[0065] I / O components 950 may include a variety of hardware components suitable for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 950 included in a particular machine will depend on the type and / or function of the machine. For example, a mobile device such as a mobile phone may include a touch input device, while a headless server or IoT device may not include such a touch input device. Fig. 9 The specific examples of I / O components illustrated in the figure are by no means limiting, and other types of components may be included in the machine 900. The grouping of the I / O components 950 is merely to simplify the discussion, and the grouping is by no means limiting. In various examples, the I / O components 950 may include a user output component 952 and a user input component 954. The user output component 952 may include, for example: a display component (e.g., a liquid crystal display (LCD) or a projector) for displaying information, an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor or a force feedback device), and / or other signal generators. The user input component 954 may include, for example, an alphanumeric input component (e.g., a keyboard or a touch screen), a pointing component (e.g., a mouse device, a touch pad, or another pointing tool), and / or a tactile input component (e.g., a physical button or touch screen that provides a position and / or a touch force or a touch gesture) configured to receive various user inputs, such as user commands and / or selections.

[0066] In some examples, I / O components 950 may include biometric components 956 and / or position components 962, as well as a variety of other environmental sensor components. Biometric components 956 may include, for example, components for detecting body expressions (e.g., facial expressions, vocal expressions, hand or body gestures, or eye tracking), measuring biosignals (e.g., heart rate or brain waves), and identifying individuals (e.g., via voice-based, retinal, and / or facial recognition). Position component 962 may include, for example, a position sensor (e.g., a global positioning system (GPS) receiver), an altitude sensor (e.g., a barometric pressure sensor from which altitude can be derived), and / or an orientation sensor (e.g., a magnetometer).

[0067] The I / O components 950 may include a communication component 964 that implements a variety of technologies operable to couple the machine 900 to network(s) 970 and / or device(s) 980 via respective communication couplings 972 and 982. The communication component 964 may include one or more network interface components or other suitable devices to interface with the network(s) 970. The communication component 964 may include, for example, components suitable for providing wired communication, wireless communication, cellular communication, near field communication (NFC), Bluetooth communication, Wi-Fi, and / or communication via other modalities. The device(s) 980 may include other machines or various peripherals (e.g., via USB coupling).

[0068] In some examples, the communication component 964 can detect the identifier or include a component suitable for detecting the identifier. For example, the communication component 964 can include a radio frequency identification (RFID) tag reader, an NFC detector, an optical sensor (e.g., a one-dimensional or multi-dimensional barcode, or other optical code) and / or an acoustic detector (e.g., a microphone for identifying an audio signal of a tag). In some examples, location information can be determined based on information from the communication component 962, such as, but not limited to: a geographic location via an Internet Protocol (IP) address, a location via Wi-Fi, cellular, NFC, Bluetooth or other wireless station identification and / or signal triangulation.

[0069] Although various embodiments have been described, the description is intended to be exemplary rather than restrictive, and it should be understood that more embodiments and implementations within the scope of the embodiments are possible. Although many possible feature combinations are shown in the drawings and discussed in this detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature of any embodiment may be used in combination with any other feature or element in any other embodiment or replace any other feature or element. Therefore, it should be understood that any of the features shown and / or discussed in this disclosure may be implemented together in any suitable combination. Therefore, unless according to the attached claims and their equivalents, the embodiments are not limited. Similarly, various modifications and changes can be made within the scope of the attached claims.

[0070] Generally, the functions described herein (e.g., in the example of FIG. 1 ) can be implemented using software, firmware, hardware (e.g., fixed logic, finite state machines, and / or other circuits), or a combination of these implementations. Figure 1-4 ). In the case of a software implementation, the program code performs the specified tasks when it is run on a processor (e.g., one or more CPUs). The program code can be stored in one or more machine-readable storage devices. The features of the technology described in this article are system-independent, which means that the technology can be implemented on a variety of computing systems with a variety of processors. For example, an implementation may include an entity (e.g., software) that causes the hardware to perform an operation, such as a processor function block, etc. For example, a hardware device may include a machine-readable medium, which may be configured to maintain instructions that cause the hardware device (including an operating system and associated hardware running thereon) to perform an operation. Therefore, the instructions can be used to configure the operating system and associated hardware to perform the operations, and thereby configure or otherwise adapt the hardware device to perform the functions described above. The instructions can be provided to the hardware element that runs the instructions by the machine-readable medium through various different configurations.

[0071] Although the foregoing has described what is considered to be the best mode and / or other examples, it is understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied in many applications, only some of which are described herein. The following claims are intended to claim any and all applications, modifications, and variations that fall within the true scope of the present teachings.

[0072] Unless otherwise indicated, all measurements, values, ratings, positions, amplitudes, sizes and other specifications set forth in this specification (including the following claims) are approximate and not exact, and are intended to have a reasonable range consistent with the functions to which they are related and the customary practices in the art to which they belong.

[0073] The scope of protection is limited only by the claims that now follow. When interpreted in light of this specification and the subsequent prosecution history, the scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims and to encompass all structural and functional equivalents. Notwithstanding the foregoing, no claim is intended to include subject matter that fails to meet the requirements of Sections 101, 102, or 103 of the Patent Act, nor should it be interpreted in such a manner. Any unexpected inclusion of such subject matter is hereby disclaimed.

[0074] Except as stated immediately above, nothing stated or described is intended or should be construed as causing any component, step, feature, object, benefit, advantage or equivalent to be conferred to the public regardless of whether it is mentioned in the claims.

[0075] It should be understood that the terms and expressions used herein have the ordinary meaning as is accorded to such terms and expressions with respect to the respective areas of inquiry and study to which they correspond unless specific meanings have otherwise been set forth herein.

[0076] Relational terms such as first and second may be used alone to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," and any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element beginning with "a" or "an" does not, without further limitation, exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0077] The abstract of the present disclosure is provided to allow the reader to quickly identify the nature of the technical disclosure. It is understood that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above detailed description, it can be seen that various features are grouped together in various examples for the purpose of simplifying the present disclosure. This method of disclosure should not be interpreted as reflecting the intention that any claim requires more features than the claims explicitly recite. On the contrary, as reflected in the attached claims, the inventive subject matter lies in less than all the features of a single disclosed example. Therefore, the attached claims are hereby incorporated into the detailed description, and each claim stands alone as a separately claimed subject matter.

Claims

1. A data processing system, comprising: processor; as well as a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the data processing system to perform the following functions: Access multiple data entries in a dataset; arranging the plurality of data entries in the data set in lexical order; placing a predetermined number of the plurality of data entries in each sub-block of the plurality of sub-blocks; performing data compression on the plurality of data entries in each of the plurality of sub-blocks to reduce redundancy in the plurality of data entries and create compressed data entries; placing one or more sub-blocks in each of the plurality of page blocks; as well as storing each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the data set, The performing of data compression on the plurality of data entries in each of the plurality of sub-blocks comprises: checking a data entry in a sub-block to determine that the data entry is not the first data entry in the sub-block; When it is determined that the data entry is not the first data entry in the sub-block: comparing the data entry to previous data entries in the sub-block to identify overlapping starting characters; and When one or more overlapping characters are identified, a number is assigned to the compressed data entry of the data entry.

2. The data processing system according to claim 1, wherein: The plurality of sub-blocks are in lexical order.

3. The data processing system according to claim 2, wherein: The plurality of page blocks are in lexical order.

4. The data processing system according to claim 1, wherein: Performing data compression on the plurality of data entries in each of the plurality of sub-blocks to create compressed data entries further comprises: Upon determining that the data entry is not the first data entry in the sub-block and comparing the data entry with the previous data entry in the sub-block to identify an overlapping starting character: assigning the number zero to a prefix of the compressed data entry when no overlapping starting characters are identified; When one or more overlapping characters are identified, assigning the number of overlapping characters to the prefix; placing in the compressed data entry a quantity for characters in the data entry other than the overlapping characters; and listing the characters excluding the overlapping characters in the compressed data entry; and A determination is made as to whether the data entry is the first data entry in a sub-block, and if so, the number of characters in the data entry is assigned as a prefix to the compressed data entry.

5. The data processing system according to claim 1, wherein: Each sub-block of the plurality of sub-blocks has a same number of data entries.

6. The data processing system according to claim 1, wherein: The plurality of sub-blocks and the plurality of page blocks comprise a main file system.

7. The data processing system according to claim 6, wherein: The executable instructions, when executed by the processor, further cause the device to perform the following functions: create a secondary data structure for temporary storage of data entries that need to be changed in the primary file system after the primary file system has been created.

8. The data processing system according to claim 7, wherein: The executable instructions, when executed by the processor, further cause the device to perform the following functions: receiving a request to perform a search for a value in the data set, the request including an ID; determining whether the ID is within a range of IDs assigned to the primary file system; Upon determining that the ID is within the range of IDs assigned to the primary file system: performing a binary search to locate the page block and the sub-block in which the value is located; and Upon locating the sub-block in which the value is located, performing a sequential search of the sub-blocks to locate the value; as well as Upon determining that the ID is outside the range of IDs assigned to the primary file system, a search of the secondary data structure is performed to locate the value.

9. A method for efficiently storing data items containing text data, comprising: Access multiple data entries in a dataset; arranging the plurality of data entries in the data set in lexical order; placing a predetermined number of the plurality of data entries in each sub-block of the plurality of sub-blocks; performing data compression on the plurality of data entries in each of the plurality of sub-blocks to reduce redundancy in the plurality of data entries and create compressed data entries; placing one or more sub-blocks in each of the plurality of page blocks; as well as storing each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the data set, The performing of data compression on the plurality of data entries in each of the plurality of sub-blocks comprises: checking a data entry in a sub-block to determine that the data entry is not the first data entry in the sub-block; When it is determined that the data entry is not the first data entry in the sub-block: comparing the data entry to previous data entries in the sub-block to identify overlapping starting characters; and When one or more overlapping characters are identified, a number is assigned to the compressed data entry of the data entry.

10. The method according to claim 9, further comprising: Upon determining that the data entry is not the first data entry in the sub-block and comparing the data entry with the previous data entry in the sub-block to identify an overlapping starting character: assigning the number zero to a prefix of the compressed data entry when no overlapping starting characters are identified; When one or more overlapping characters are identified, assigning the number of overlapping characters to the prefix; placing in the compressed data entry a quantity for characters in the data entry other than the overlapping characters; as well as listing the characters excluding the overlapping characters in the compressed data entry; as well as A determination is made as to whether the data entry is the first data entry in a sub-block, and if so, the number of characters in the data entry is assigned as a prefix to the compressed data entry.

11. The method according to claim 10, wherein: The plurality of sub-blocks and the plurality of page blocks comprise a main file system.

12. The method of claim 11, further comprising creating a secondary data structure for temporary storage of data entries that need to be changed in the primary file system after the primary file system has been created.

13. The method according to claim 12, further comprising: receiving a request to perform a search for a value in the data set, the request including an ID; determining whether the ID is within a range of IDs assigned to the primary file system; Upon determining that the ID is within the range of IDs assigned to the primary file system: performing a binary search to locate the page block and the sub-block in which the value is located; Upon locating the sub-block in which the value is located, performing a sequential search of the sub-blocks to locate the value; as well as Upon determining that the ID is outside the range of IDs assigned to the primary file system, a search of the secondary data structure is performed to locate the value.

14. A computer readable medium having stored thereon instructions which, when executed, cause a programmable device to: Access multiple data entries in a dataset; arranging the plurality of data entries in the data set in lexical order; placing a predetermined number of the plurality of data entries in each sub-block of the plurality of sub-blocks; performing data compression on the plurality of data entries in each of the plurality of sub-blocks to reduce redundancy in the plurality of data entries and create compressed data entries; as well as placing one or more sub-blocks in each of the plurality of page blocks; and storing each of the plurality of page blocks in a storage device to provide efficient searching and improved functionality for the data set, The performing of data compression on the plurality of data entries in each of the plurality of sub-blocks comprises: checking a data entry in a sub-block to determine that the data entry is not the first data entry in the sub-block; When it is determined that the data entry is not the first data entry in the sub-block: comparing the data entry to previous data entries in the sub-block to identify overlapping starting characters; and When one or more overlapping characters are identified, a number is assigned to the compressed data entry of the data entry.

15. The computer-readable medium of claim 14, wherein: The instructions further cause the programmable device to: Upon determining that the data entry is not the first data entry in the sub-block and comparing the data entry with the previous data entry in the sub-block to identify an overlapping starting character: assigning the number zero to a prefix of the compressed data entry when no overlapping starting characters are identified; When one or more overlapping characters are identified, assigning the number of overlapping characters to the prefix; placing in the compressed data entry a quantity for characters in the data entry other than the overlapping characters; as well as listing the characters excluding the overlapping characters in the compressed data entry; as well as A determination is made as to whether the data entry is the first data entry in a sub-block, and if so, the number of characters in the data entry is assigned as a prefix to the compressed data entry.

Images