Adjustable inverted index

Abstract

Example methods and systems are directed to reverse indexing. According to some examples, a reverse index is generated based on source data and a ranking list threshold. The reverse index includes one or more restricted emission lists. Each restricted emissions list has a maximum size corresponding to an emissions list threshold. The method may include receiving a search query including a value identifying a restricted emission list of the one or more restricted emission lists. The value may be used to retrieve and return one or more record identifiers from the identified restricted emissions list. The record identifier uniquely identifies one of a plurality of records in the source data.

Description

Technical Field

[0001] The subject matter disclosed herein generally relates to data searching. More specifically, the subject matter relates to the generation of inverted indices and search operations facilitated by inverted indices. Background Art

[0002] In a conventional database, data may be arranged in a tabular format, with each row representing a record and each column representing an attribute. An inverted index is a data structure that maps unique values ​​within a database to identifiers of corresponding records (e.g., rows) in which those values ​​appear. This data structure can efficiently track instances of values ​​in an underlying database or document collection (referred to herein as "source data"). In use, an inverted index can quickly identify records containing desired values, rather than performing a full scan of the source data to identify records containing a particular value (e.g., by checking every entry in a particular column). Therefore, for certain access patterns, an inverted index can speed up the search process, saving computing cycles and processing power.

[0003] Most usefully, inverted indices are typically stored in main memory, such as random access memory. In addition to the memory already required to store the source data, inverted indices consume a large amount of main memory. Furthermore, synchronization and maintenance of inverted indices can be resource intensive. Therefore, it would be technically beneficial to reduce resource requirements, and in particular memory requirements, associated with inverted indices. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] In the accompanying drawings, some examples are shown for purposes of illustration and not limitation. In the drawings, which are not necessarily drawn to scale, the same numerals may describe similar components in different views or examples. To more easily identify the discussion of any particular element or action, the most significant digit or digits in the reference numerals refer to the figure number in which the element is first introduced.

[0005] Figure 1 is an illustration of a network environment suitable for generating and adjusting an inverted index and retrieving stored data according to some examples.

[0006] Figure 2 is a block diagram of certain components of an indexing and searching system according to some examples.

[0007] Figure 3 is a flow diagram illustrating the operation of a method suitable for generating an inverted index having one or more restricted emission lists, according to some examples.

[0008] Figure 4 Source data represented in table form according to some examples is shown.

[0009] Figure 5An inverted index according to some examples is shown, and emission list thresholds are further conceptually illustrated.

[0010] Figure 6 An inverted index according to some examples is shown, and further conceptually illustrating an emission list threshold and record identifiers omitted from the inverted index based on the emission list threshold.

[0011] Figure 7 is a flow diagram illustrating the operation of a method suitable for retrieving data using an inverted index according to some examples.

[0012] Figure 8 is a block diagram illustrating a software architecture of a computing device according to some examples.

[0013] Fig. 9 is a block diagram of a machine in the form of a computer system according to some examples in which instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0014] Example methods and systems are directed to dynamically configurable or dynamically adjustable inverted indexes. The term "source data" as used in this disclosure refers to underlying data or base data used to generate an inverted index. Examples of source data include a table, a collection of tables, a collection of tokenized data, or a collection of documents. In some cases, as further described below, the source data is preprocessed before generating an inverted index.

[0015] The source data includes a plurality of records. As used herein, the term "record" refers to a single uniquely identifiable entry in the source data, such as a row in a table containing values ​​of columns of one or more tables. The term "record identifier" refers to any suitable identifier for a record, such as an identifier for a row.

[0016] The inverted index includes one or more posting lists. The term "posting list" as used in the present disclosure refers to a list or set of postings that identify where a specific value (also referred to as a "key" in the context of an inverted index) appears in the source data. Each posting may include a record identifier and, in some cases, a payload such as a frequency indicator. Thus, the posting list serves as a reference to the original record of the source data. As described below, examples of the present disclosure enable the use of restricted posting lists within an inverted index to reduce the memory footprint of the inverted index.

[0017] According to some examples, a method includes accessing source data including a plurality of records. Prior to generating an inverted index, or prior to completing the generation of the inverted index, an emission list threshold is identified. The emission list threshold defines the maximum size of one or more emission lists of the inverted index. The maximum size is, for example, the maximum number of emissions that can be included in a particular emission list of the inverted index. The term "restricted posting list" used in the present disclosure refers to an emission list that is limited, constrained, or capped by an emission list threshold. For example, although a particular value may appear in ten different records within a source data set, a restricted emission list may have five emission list thresholds, resulting in only five of the ten records being identified by the restricted emission list within the generated inverted index.

[0018] The method may include generating an inverted index based on the source data and the emission list threshold. The inverted index generated in this manner then includes one or more restricted emission lists. As described above, each restricted emission list has a maximum size corresponding to the emission list threshold. Therefore, in some examples, the emission list can be truncated by implementing the emission list threshold to reduce memory requirements.

[0019] Emission list thresholds may be applied to one or more emission lists, and may be applied globally (e.g., across the entire inverted index) or locally (e.g., per table or per attribute). In some examples, an inverted index may have a global emission list threshold that applies to all emission lists of the inverted index. In other examples, only a subset of emission lists may have emission list thresholds, or different emission lists within the same inverted index may have different emission list thresholds. In this way, an inverted index may be defined or configured in a flexible, customizable, or granular manner.

[0020] Generation of the inverted index may include, for each restricted emission list, populating the restricted emission list with emissions (eg, record identifiers from source data) until an emission list threshold has been reached, or until all possible emissions have been included, whichever occurs first.

[0021] In some examples, the method includes generating a flag (e.g., a Boolean indicator) that indicates whether an emissions list threshold has been reached for a restricted emissions list. The flag may indicate, for example, that the emissions list threshold has been reached and that the source data includes additional record identifiers that have been omitted from the restricted emissions list. In short, the flag may indicate that "there is more" relevant data within the source data. The flag is stored in association with the corresponding restricted emissions list.

[0022] In some examples, the emission list threshold can be adjusted before or after the generation or initialization of the inverted index. For example, user input can be received to define, select, or adjust the emission list threshold. Adjustment of the emission list threshold may cause the related restricted emission list to be dynamically updated.

[0023] Systems according to some examples can be configured to automatically adjust emission list thresholds. For example, the system can analyze source data and determine appropriate emission list thresholds based on patterns or attributes of the source data. As another example, the system can monitor memory consumption associated with the inverted index and dynamically adjust the emission list threshold, for example, adaptively reducing the emission list threshold to reduce the memory footprint of the inverted index. Automatic adjustment can be performed at different levels, for example, globally, by table, or by column.

[0024] The techniques described herein may also include receiving a search query, and in response to receiving the search query, retrieving emissions, such as record identifiers, from one or more emission lists of the inverted index. The search query may include a value that identifies a particular restricted emission list of the inverted index. For example, the search query may match a key of the restricted emission list in whole or in part. The value in the search query may be used to retrieve and return one or more record identifiers from the identified restricted emission list.

[0025] As described above, an inverted index can allow for fast lookups compared to conventional table scans. However, a restricted emissions list may not identify all records containing a particular value or key due to an emissions list threshold. A search operation may include detecting whether an emissions list threshold has been reached with respect to an identified restricted emissions list, or detecting whether the source data includes record identifiers that have been omitted from the restricted emissions list due to an emissions list threshold. A fallback search may be performed to retrieve record identifiers omitted from the restricted emissions list.

[0026] For example, the search operation may include checking a flag associated with the identified restricted emissions list, and based on the flag, identifying that the source data includes relevant records that are not identified by the restricted emissions list due to restrictions on the number of emissions in the restricted emissions list. In this case, the flag may be considered a "fallback search flag." Optionally, in response to detecting the flag, the source data may be checked to identify whether there are other relevant records to search.

[0027] The method may include performing a fallback search in response to detecting a fallback search flag, in response to detecting that an emission list threshold has been reached, in response to detecting that there are other relevant records to search, or a combination thereof. The term "fallback search" as used in this disclosure refers to a search or scan performed in the source data (as opposed to an inverted index) to retrieve one or more entries (e.g., record identifiers) that cannot be retrieved from the restricted emission list due to the above-mentioned size limit.

[0028] Therefore, examples of the present disclosure enable reducing the memory footprint of an inverted index through an adjustable or customizable data structure. Since all information represented in the inverted index can also be represented in the associated source data, the information omitted from the adjusted inverted index due to the implementation of the emission list threshold remains able to be retrieved via the source data. In this way, the system according to the present invention can enable users to achieve a desired balance between memory utilization and search performance. For example, in a specific implementation, it may be known that most search queries can be processed by considering only the latest (most recent) 50% of the records in the database. The inverted index can then be configured to fill its restricted emission list in reverse chronological order with a predetermined number of emissions from the records in the source data, thereby maximizing the probability that the inverted index can process the search query while reducing memory consumption. In some examples, when compared to using a conventional inverted index based on a source data set, memory consumption can be significantly reduced by utilizing an inverted index with one or more suitable emission list thresholds associated with the same source data set.

[0029] Examples of the present disclosure provide a useful tool that enables control of the balance between memory and retrieval performance at a granular level. As described herein, an adjusted inverted index can allow for reduced storage costs (e.g., by reducing main memory consumption), smaller compute instances, or more dynamic scaling. As described above, emission list thresholds can be automatically adjusted, which can improve efficiency and reduce the level of manual input required to adjust or manage index data.

[0030] When considering the effects of the present disclosure as a whole, one or more of the methods described herein can eliminate the need for certain efforts or resources that would otherwise be involved in a database or data retrieval system. For example, due to the reduced memory usage caused by the inverted index, the computing resources utilized by the system, database, or network can be more efficiently utilized or reduced. Memory freed up in this manner can be utilized by other objects, applications, or resources. Alternatively or additionally, the resulting free memory can be put to "sleep," suspended, or powered off, allowing the memory to use less energy. In addition, by intelligently selecting or adjusting emission list thresholds, a reduction in memory requirements can be achieved without significantly hampering performance, retrieval time, or overall index usefulness.

[0031] Figure 1 1 is an illustration of a networked computing environment 100 in which some examples of the present disclosure may be implemented or deployed. One or more servers in a server system 104 provide server-side functionality via a network 102 to networked devices, an example of which is a user device 106 accessed by a user 128. A web client 110 (e.g., a browser) or a programmatic client 108 (e.g., an "app") may be hosted and executed on the user device 106.

[0032] Application program interface (API) server 118 and web server 120 provide respective programming and web interfaces to the components of server system 104. Specific application server 116 hosts indexing and search system 122, which includes components, modules or applications that help users create and dynamically adjust inverted indices and perform search operations facilitated by such inverted indices.

[0033] User device 106 may communicate with application server 116, for example, via a web interface supported by web server 120 or via a programming interface provided by API server 118. Figure 1 Only a single user device 106 is shown in FIG, but in some examples, multiple user devices may be communicatively coupled to the server system 104. In addition, while certain functions may be described herein as being performed at a user device 106 (e.g., a web client 110 or a programming client 108) or a server system 104, the location of certain functions within a user device 106 or a server system 104 may be a design choice.

[0034] Application server 116 is communicatively coupled to database server 124, facilitating access to one or more information repositories, such as database 126. In some examples, database 126 includes a storage device that stores information to be processed or transmitted by index and search system 122, such as source data for generating an inverted index.

[0035] The application server 116 accesses application data (e.g., application data stored by the database server 124) to provide one or more applications or tools to the user device 106 via the web interface 130 or the application interface 132. For example, as shown below by way of example and with specific reference to Figure 2-Figure 7 As further described, application server 116 using indexing and search system 122 may provide functionality that enables user 128 to define, build, adjust, and use a variety of data structures, including inverted indexes.

[0036] To access the tools provided by the indexing and search system 122, a user 128 may create an account (or access an existing account) with an entity (e.g., a service provider) associated with the server system 104. The user 128 may use the account credentials to access the web interface 130 (via a suitable web browser) and request access to one or more tools. In some examples, the user 128 may also access the tools using a dedicated programming client 108, in which case some functionality may be provided on the client side, while other functionality may be provided on the server side.

[0037] In some examples, application server 116 is part of a cloud-based platform provided by an entity associated with server system 104 that allows account holders to take advantage of various tools of index and search system 122. For example, an account holder can create, manage, and use retrieval functions by accessing one or more cloud instances. Index and search system 122 can provide a user interface that facilitates communication between user 128 and index and search system 122 or between different users of index and search system 122.

[0038] In some examples, application server 116 provides a set of features to user 128 that enables user 128 to build, manage, and scale search solutions (e.g., for a website or application). The services provided by index and search system 122 may provide a high-level interface for creating and managing search solutions, including defining key parameters of an inverted index, without requiring user 128 to configure or deal with low-level details of the solution, such as the actual indexing of data.

[0039] One or more of the application server 116, the database server 124, the API server 118, the web server 120, and the indexing and searching system 122 may be implemented in whole or in part in a computer system, as described below with reference to Fig. 9 In some examples, a third-party application, such as a third-party application 114 executing on a third-party server 112, can communicate with an application server 116 via a programming interface provided by an API server 118. For example, a third-party application can support one or more features or functions on a website or platform hosted by a third party, or can execute certain methods and provide input or output information to the application server 116 for further processing or publication. For example, a third-party server 112 can host source data used by a user 128 in creating a dynamically adjustable inverted index.

[0040] The network 102 may be any network that enables communication between machines, databases, and devices. Thus, the network 102 may be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The network 102 may include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0041] Figure 2 is a block diagram illustrating certain components of the indexing and search system 122 according to some examples. The indexing and search system 122 is shown to include a communication component 202, a source data component 204, a threshold component 206, an index generation component 208, a tagging component 210, an index adjustment component 212, and a search component 214.

[0042] In some examples, Figure 2 At least some of the components shown in are configured to communicate with each other to implement the aspects described herein. Any one or more components described herein can be implemented using hardware (e.g., one or more processors of one or more machines) or a combination of hardware and software. For example, any component described herein can be implemented by a processor configured to perform the operation described herein for the component. In addition, any two or more of these components can be combined into a single component, or the functions described herein for a single component can be subdivided in multiple components. In addition, according to various examples, the components described herein can be implemented using a single machine, database, or device, or distributed on multiple machines, databases, or devices.

[0043] The communication component 202 enables a user 128 of a user device 106 to send data to, and receive data from, the indexing and search system 122. The communication component 202 may provide one or more user interfaces to enable the user 128 to utilize the tools of the indexing and search system 122 to, for example, create or manage indexes, create and manage search queries, or perform information retrieval.

[0044] The source data component 204 is responsible for managing source data, such as one or more data tables used as underlying data associated with one or more inverted indexes. The source data component 204 can communicate with a storage component (eg, database 126) to store and maintain source data.

[0045] The threshold component 206 is configured to enable the use of restricted emission lists within the inverted index. As described above, the user 128 may, for example, provide user input to set or select one or more emission list thresholds for the inverted index. Alternatively or additionally, the threshold component 206 may perform automatic selection of one or more emission list thresholds.

[0046] The index generation component 208 is responsible for creating an inverted index. The index generation component 208 can, for example, perform tokenization or other pre-processing operations (such as stop word elimination, special character removal or case adjustment) to generate the required source data. In order to create an index, the index generation component 208 can create an emission list for each value in the source data (for example, each unique attribute value or tag). In the case where the emission list is not associated with the emission list threshold, the emission list can be filled in a conventional manner, for example, by adding an inverted row to identify each occurrence of the relevant value in the source data. In the case where the emission list is a restricted emission list (linked to the emission list threshold), the index generation component 208 can also fill the emission list with emissions, provided that the number of emissions cannot exceed the emission list threshold. Note that multiple types of emission list thresholds can be implemented. In some cases, the emission list threshold can even be set to zero, which can be referred to as a "never store" setting, causing the index generation component 208 to not store entries related to a specific value or key. Examples of different methods for filling a restricted emission list are provided below.

[0047] The marking component 210 is configured to generate and store a flag indicating restriction details about a particular inverted index. For example, the marking component 210 can store a flag in the form of a Boolean value ("yes" or "no") for each restricted emissions list, indicating whether an emissions list threshold has been reached, for example, indicating whether the emissions list has been "fully filled". The flag can be a bit or Boolean field that can be turned on or off to indicate a particular state. The flag can also indicate whether additional records that do not "fit" the restricted emissions list are available or retrievable from the source data. The flag can be a fallback search flag that indicates that a fallback search is required to retrieve further data from the source data (in addition to a lookup search in the inverted index). As will be described in more detail below, the use of one or more flags can facilitate retrieval of search results during a search operation.

[0048] Flags change over time, and the marking component 210 can be used to maintain flags automatically or through some degree of user input. For example, the inverted index may not initially include any fallback search flags, but as more records are added to the source data and the inverted index is updated, the marking component 210 can add fallback search flags. For example, flags can be maintained by performing "in-place" updates. This can involve modifying flags directly in the database without creating a copy of the data. For example, another maintenance method is a "copy and switch" update, which involves creating a new updated copy and then switching the database to use the new copy after the update is completed. In some examples, the "in-place" technique can execute faster, while the "copy and switch" technique can avoid potential data inconsistencies associated with in-place updates.

[0049] The inverted index and any associated flags can be stored in a variety of ways, depending on user needs or usage. For example, the inverted index can be stored in main memory or written to disk. The inverted index can be stored in a format that allows fast lookups, such as a simple array, a hash map, or an A- or B-tree.

[0050] The index adjustment component 212 is responsible for causing the adjustment of the emission list threshold, and the resulting change of the inverted index. As described above, user input can be received (e.g., via the communication component 202) to adjust the emission list threshold, or the index adjustment component 212 can automatically adjust the emission list threshold to meet certain requirements, such as optimizing memory usage, optimizing performance, or balancing memory usage and performance. Adjustments to the emission list threshold may cause the associated restricted emission list to be dynamically updated. For example, if the emission list threshold of a particular restricted emission list is adjusted from five to ten, additional emissions can be automatically added to the emission list threshold, so that more record identifiers (records in the reference source data) that did not previously form part of the restricted emission list become part of the "extended" restricted emission list.

[0051] The search component 214 is configured to process the search query. The search component 214 is responsible for performing an inverted index lookup and a fallback search in the source data. In some examples, due to an emission list threshold applied to the inverted index or a particular emission list of the inverted index, it may not be possible to retrieve a complete set of search results by querying the inverted index alone. In other words, due to the size cap, the relevant records of the keys matching the restricted emission list are omitted from the restricted emission list. In this case, as further described below, the search component 214 can perform a fallback search within the source data.

[0052] Figure 3 is a flow chart illustrating the operation of a method 300 suitable for generating an inverted index having one or more restricted emission lists according to some examples. By way of example and not limitation, aspects of the method 300 may be performed by Figure 1 and Figure 2 In addition, to illustrate certain aspects of method 300, reference is made to Figure 4-Figure 6 An exemplary data structure is shown.

[0053] The method 300 begins at an open loop element 302 and proceeds to operation 304 where the index and search system 122 accesses source data including a plurality of records. Figure 4 Example source data in the form of table 400 is shown. Table 400 includes eight records, each represented by a row. Table 400 includes two columns, defining a record identifier 402 (eg, row ID) and a value of an attribute 404 for each row. Figure 4 , attribute 404 is "city". Figure 4 In the source data, records are arranged in chronological order. For example, row "8" represents the most recent record added to the source data.

[0054] It should be understood that table 400 is shown to provide a simple example for the purpose of illustrating certain aspects of the present disclosure. The source data may be represented or stored in different formats or different structures, and in other examples, may include a large number of records, multiple attributes, multiple tables, etc.

[0055] The method 300 proceeds to operation 306, where an emission list threshold is identified. As described above, depending on the implementation, the emission list threshold may be user-defined or may be automatically set by the index and search system 122. The index and search system 122 may analyze the records in the table 400 and determine an appropriate emission list threshold based on the analysis, for example, by considering certain requirements, such as reducing the memory footprint of the inverted index.

[0056] In some examples, the emission list threshold is a global emission list threshold that is applied to each emission list of the inverted index that will be generated from table 400, e.g., each emission list will be a restricted emission list. In other examples, the emission list threshold may be an attribute-specific emission list threshold, e.g., a threshold that applies only to emission lists for specific attribute values ​​in the source data. In this case, the restricted emission lists in the inverted index may be a subset of the total number of emission lists, as some emission lists may not have an emission list threshold. For example, user 128 may place more importance on the first column of a table than on the second column of the table, and therefore may wish to generate a full, "regular" inverted index for the values ​​in the first column, while wishing to reduce total storage by generating a truncated index for the values ​​in the second column.

[0057] As another example, where the source data includes multiple tables, the user 128 may place greater emphasis on a particular table. In this case, the emission list threshold may be a table-specific emission list threshold that applies only to values ​​that appear in the particular table.

[0058] Thus, the emission list threshold may define a maximum size associated with one or more emission lists. In the examples described below, the emission list threshold is a global emission list threshold that applies to the entire inverted index. Referring to Table 400, the emission list threshold may be set to, for example, "five" or "two". Figure 5 and Figure 6 These two examples are further described.

[0059] Turn again Figure 3, the method 300 proceeds to operation 308, where the index and search system 122 identifies a predetermined order for populating the inverted index. For example, the predetermined order may be a reverse chronological order, where the newest (most recently added) records of the table 400 are added to the inverted index first. The reverse chronological order is used as described below. Figure 5 and Figure 6 Optionally, the predetermined order may be a time order. It should be understood that different orders or filling rules may be applied depending on the specific usage.

[0060] At operation 310, the indexing and search system 122 begins generating an inverted index from the source data in table 400. The inverted index is populated based on a predetermined order (operation 312). An emission list is generated to represent each unique value in table 400 ("Berlin", "Hamburg", "Paris", and "Sydney"). Record identifiers are added to each emission list until all possible record identifiers are added, or until an emission list threshold is reached.

[0061] refer to Figure 5 , a first example of an inverted index 500 is shown. Inverted index 500 is generated based on table 400 using an emission list threshold of five. Since all values ​​appear less than five times in the source data, the emission list threshold is not reached for any emission list, and inverted index 500 is similar to a conventional inverted index. In other words, the emission list threshold does not result in any storage savings compared to a standard inverted index, and retrieval performance will also reflect that of a standard inverted index.

[0062] Reference now Figure 6 , a second example of an inverted index 600 is shown. Inverted index 600 is also generated based on table 400, but uses an emission list threshold of 2. The values ​​"Hamburg", "Paris", and "Sydney" appear no more than twice in table 400, so all related record identifiers are included in the inverted index. For the value "Sydney", the emission list reaches its emission list threshold, but record identifiers for records that do not contain "Sydney" are omitted from inverted index 600.

[0063] On the other hand, for the value "Berlin" in inverted index 600, this value appears in four different records in table 400, and therefore only record identifiers for two of the records may be included in emission list 602 (e.g., Figure 6). As described above, the emission list 602 is populated using a reverse chronological order, and the first record identifier 604 "8" is added first because the corresponding record was most recently added to the table 400. Continuing the reverse chronological order, the second record identifier 606 "6" is added to the emission list 602. This results in the emission list threshold being reached, and no further emissions can be added to the emission list 602. Therefore, the two record identifiers "1" and "4" are excluded. The excluded record identifier 608 is in Figure 6 It is conceptually shown in FIG.

[0064] Therefore, in Figure 6 In the case of , the inverted index 600 is different from a conventional inverted index in that the emission list 602 is truncated. Figure 5 Compared with the inverted index 500 , this reduces the memory usage of the inverted index 600 .

[0065] As described above, the inverted index 600 has a global emission list threshold, which means that all of its emission lists are restricted emission lists. However, this does not mean that all emission lists must meet the global emission list threshold. At operation 314, the method 300 includes generating a flag for each emission list. For example, the indexing and search system 122 can apply a Boolean flag that indicates for each emission list whether the emission list threshold has been met. In this case, Figure 6 In the case of , the flag for "Berlin" and "Sydney" would be "yes" (and the flag for "Hamburg" and "Paris" would be "no"). Alternatively, the flag may only indicate "yes" if the emission list threshold has been met and there is an omitted record identifier associated with the value in question. In this case, the flag may be referred to as a fallback search flag and is not used in Figure 6 , only the flag for "Berlin" is "yes", while all other flags are "no". In other words, for queries related to the value / key "Berlin", the fallback search flag provides a quick indicator that the source data must be searched to locate the excluded record identifier 608.

[0066] Figure 3 The method 300 proceeds to block 316, where the inverted index is stored. For example, the inverted index 600 and the flag (or flags) may be associated with a pointer to the original source data (e.g. Figure 4 The pointer of the table 400 of the index is stored in the main memory, and the original source data can be stored in the auxiliary memory. Therefore, reducing the memory footprint of the inverted index by applying one or more emission list thresholds can lead to more efficient main memory usage. The method 300 ends at the closed loop element 318.

[0067] Reference again Figure 6In use, the inverted index 600 can be updated as the source data is updated. For example, new rows "9" and "10" can be added to the table 400, both of which have the value "Berlin" in the "City" column. This will cause the index and search system 122 to automatically replace "6" and "8" in the emission list 602 with "9" and "10" according to the popular reverse chronological order. Similarly, if another row "11" with the value "Sydney" is added to the table 400, this will cause the record "5" to be removed from the "Sydney" emission list by the index and search system 122 and automatically replaced by the record "11". In some examples, the inverted index can therefore be dynamically updated to reflect changes in the source data, for example, to include records that were recently added to the source data within the inverted index.

[0068] Still reference Figure 6 , the inverted index 600 can be adjusted by dynamically changing the emission list threshold. For example, in order to reduce memory consumption, the emission list threshold can be changed from "two" to "one", resulting in the emission lists of "Berlin" and "Sydney" both "missing" one emission.

[0069] Optionally, to improve search performance, the emission list threshold may be changed to "three" so that additional emissions are added to the emission list 602. Thus, the maximum size of each restricted emission list may be adjusted to correspond to the adjusted emission list threshold. As described above, the balance or tradeoff between memory requirements and performance requirements may be dynamically selected or adjusted manually by the user 128 or automatically by the indexing and search system 122.

[0070] It should be understood that data can be stored in a variety of formats or structures. For example, the emission list can be stored as an array, in which case a copy operation may be required to remove entries (e.g., the "oldest" entry). Other formats may include linked lists, ring buffers, and other related formats. Similarly, a variety of techniques can be used to change the inverted index. For example, the inverted index can be locked by the index and search system 122 so that the relevant columns (emission lists) can be updated, such as by adding more entries as needed.

[0071] Figure 7 is a flowchart illustrating the operation of a method 700 suitable for retrieving data using an inverted index according to some examples. By way of example and not limitation, aspects of the method 700 may be Figure 1 and Figure 2 The components, devices, systems or databases shown in the following are performed. Figure 4 Table 400 and Figure 6 The inverted index 600 and Figure 7 The operations of are used to illustrate certain aspects of method 700.

[0072] Method 700 begins at an open loop element 702 and proceeds to operation 704, where the index and search system 122 receives a search query including a value. Figure 6 In the case of inverted index 600 , a search query may include the value “Berlin” or the value “Hamburg”.

[0073] At operation 706, the index and search system 122 matches the value to a key in the inverted index 600. For example, where the value is "Berlin", the index and search system 122 matches the search query to the relevant key and the emission list 602. The method 700 then proceeds to operation 708, where the index and search system 122 retrieves the emission (e.g., record identifier) ​​from the relevant emission list. For example, the index and search system 122 retrieves the first record identifier 604 and the second record identifier 606 from the emission list 602 in response to the "Berlin" query.

[0074] The index and search system 122 checks a flag associated with the relevant sorted list to determine whether a fallback search is required (decision operation 710). For example, if the search query is for "Berlin", the inverted index 600 has a fallback search flag that essentially indicates that "there is more to find" within the source data. In this case, the method 700 proceeds to operation 712, where the index and search system 122 searches the table 400 to identify the excluded record identifiers 608, thereby obtaining a complete set of search results. On the other hand, if the search query is for "hamburgers", then based on the flag, the index and search system 122 determines that a fallback search is not required (because the sorted list in the inverted index 600 already provides a complete set of search results).

[0075] Therefore, in the case where the limited or truncated inverted index cannot fully "answer" the request, a fallback search can be performed. In some examples, the indexing and search system 122 can narrow the scope of the fallback search by analyzing the contents of the relevant emission list, thereby optimizing the fallback search. Specifically, the indexing and search system 122 can only identify a range or subset of records in the source data that are not covered by the emission list, and only search the range or subset of records, rather than performing a full scan of the source data.

[0076] For example, in the case of "Berlin", the index and search system 122 detects that the emission list 602 includes record identifiers "6" and "8", and based on this detection, determines that it only needs to scan records "1" through "5" to search for the value "Berlin" in the source data (e.g., table 400). In this way, entries that have already been processed or covered within the inverted index can be skipped, thereby improving the efficiency of the search operation and reducing the processing resources required for fallback searches.

[0077] At operation 714, the index and search system 122 then returns the retrieved record identifiers. For example, in the case of the query "Berlin", the index and search system 122 can return the combined results of the inverted index lookup (returning "6" and "8") and the fallback search (returning the missing records "1" and "4"). The method 700 ends at the closing loop element 716.

[0078] It will be appreciated that an inverted index generated using the techniques described herein can be used to handle more complex queries than the single-value queries involved in the above examples. For example, if a search query includes two values, each associated with a different emission list, the index and search system 122 can intersect the emission list to find all records where both values ​​appear, if any. In some examples, a combination of an inverted index lookup and a fallback search can be used for this purpose.

[0079] In view of the above-mentioned embodiments of the subject matter, the present application discloses the following list of examples, wherein one feature of a single example or more than one feature of an example, combined and optionally combined with one or more features of one or more other examples, are other examples that also fall within the scope of the disclosure of the present application.

[0080] Example 1 is a system comprising: at least one memory storing instructions; and one or more processors configured by the instructions to perform operations, including: generating an inverted index including one or more restricted emission lists based on source data and an emission list threshold, each restricted emission list having a maximum size corresponding to the emission list threshold, and the source data including multiple records; receiving a search query, the search query including a value that identifies a restricted emission list in the one or more restricted emission lists; in response to receiving the search query, retrieving one or more record identifiers from the identified restricted emission lists using the value, each record identifier uniquely identifying one of the multiple records; and returning one or more record identifiers.

[0081] In Example 2, the subject matter of Example 1 includes, wherein generating an inverted index includes, for each restricted emission list: populating the restricted emission list with at least one record identifier; generating a flag indicating whether an emission list threshold has been reached with respect to the restricted emission list; and storing the flag associated with the restricted emission list.

[0082] In Example 3, the subject matter of Example 2 includes, in response to receiving a search query: detecting that a fallback search is needed based at least in part on the flag; and in response to detecting that a fallback search is needed, using the value to perform a fallback search to retrieve from the source data at least one record identifier that cannot be retrieved from the identified restricted emissions list.

[0083] In Example 4, the subject matter of Example 3 includes, wherein performing the fallback search includes: identifying a range or subset of records in the source data that are not covered by the identified restricted emissions list; and searching the range or subset of records to identify a record that includes the value.

[0084] In Example 5, the subject matter of Examples 1-4 includes, wherein generating the inverted index comprises populating each restricted emission list with record identifiers of records in the source data until an emission list threshold is reached.

[0085] In Example 6, the subject matter of Example 5 includes, wherein each restricted emissions list is populated based on a predetermined order.

[0086] In Example 7, the subject matter of Example 6 includes, wherein the predetermined order is reverse chronological order.

[0087] In Example 8, the subject matter of Examples 1-7 includes, wherein the inverted index includes a plurality of emission lists, and wherein the emission list threshold is a global emission list threshold applied to each emission list of the inverted index such that each emission list is a restricted emission list.

[0088] In Example 9, the subject matter of Examples 1-8 includes, wherein the inverted index includes multiple emission lists, wherein the source data includes values ​​of multiple attributes, and wherein the emission list threshold is an attribute-specific emission list threshold applied to one or more restricted emission lists, the one or more restricted emission lists being a subset of the multiple emission lists.

[0089] In Example 10, the subject matter of Examples 1-9 includes, wherein the inverted index includes multiple emission lists, wherein the source data includes multiple tables, and wherein the emission list threshold is a table-specific emission list threshold applied to one or more restricted emission lists, the one or more restricted emission lists being a subset of the multiple emission lists.

[0090] In Example 11, the subject matter of Examples 1-10 includes receiving user input to select an emissions listing threshold.

[0091] In Example 12, the subject matter of Examples 1-11 includes: analyzing records in the source data; and selecting an emissions listing threshold based on analyzing the records in the source data.

[0092] In Example 13, the subject matter of Examples 1-12 includes, after generating an inverted index: receiving user input to adjust an emission list threshold; and in response to receiving user input to adjust the emission list threshold: defining an adjusted emission list threshold, and adjusting the maximum size of each restricted emission list to correspond to the adjusted emission list threshold.

[0093] In Example 14, the subject matter of Examples 1-13 includes, wherein the source data includes at least one table, wherein records are represented as corresponding rows in the at least one table, each table further includes at least one column, each column represents an attribute, wherein unique values ​​of the attributes are used as corresponding keys in the inverted index.

[0094] Example 15 is a method comprising: generating an inverted index comprising one or more restricted emission lists based on source data and an emission list threshold, each restricted emission list having a maximum size corresponding to the emission list threshold, and the source data comprising multiple records; receiving a search query, the search query comprising a value identifying a restricted emission list in the one or more restricted emission lists; in response to receiving the search query, retrieving one or more record identifiers from the identified restricted emission lists using the value, each record identifier uniquely identifying one of the multiple records; and returning one or more record identifiers.

[0095] In Example 16, the subject matter of Example 15 includes, wherein generating the inverted index includes, for each restricted emissions list: populating the restricted emissions list with at least one record identifier; generating a flag indicating whether an emissions list threshold has been reached with respect to the restricted emissions list; and storing the flag associated with the restricted emissions list.

[0096] In Example 17, the subject matter of Example 16 includes, in response to receiving a search query: detecting that a fallback search is needed based at least in part on the flag; and in response to detecting that a fallback search is needed, using the value to perform a fallback search to retrieve at least one record identifier from the source data that cannot be retrieved from the identified restricted emissions list.

[0097] Example 18 is a non-temporary computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: generating an inverted index comprising one or more restricted emission lists based on source data and an emission list threshold, each restricted emission list having a maximum size corresponding to the emission list threshold, and the source data comprising multiple records; receiving a search query, the search query comprising a value that identifies a restricted emission list in the one or more restricted emission lists; in response to receiving the search query, retrieving one or more record identifiers from the identified restricted emission lists using the value, each record identifier uniquely identifying one of the multiple records; and returning one or more record identifiers.

[0098] In Example 19, the subject matter of Example 18 includes, wherein generating the inverted index includes, for each restricted emissions list: populating the restricted emissions list with at least one record identifier; generating a flag indicating whether an emissions list threshold has been reached with respect to the restricted emissions list; and storing the flag associated with the restricted emissions list.

[0099] In Example 20, the subject matter of Example 19 includes, in response to receiving a search query: detecting a need for a fallback search based at least in part on a flag; and in response to detecting a need for a fallback search, using the value to perform a fallback search to retrieve from the source data at least one record identifier that cannot be retrieved from the identified restricted emissions list.

[0100] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-20.

[0101] Example 22 is an apparatus comprising components for implementing any of Examples 1-20.

[0102] Example 23 is a system implementing any of Examples 1-20.

[0103] Example 24 is a method of implementing any of Examples 1-20.

[0104] Figure 8 8 is a block diagram 800 illustrating a software architecture 802 of a computing device according to some examples. The software architecture 802 can be used in conjunction with a variety of hardware architectures, for example, as described herein. Figure 8 This is merely a non-limiting illustration of a software architecture, and many other architectures may be implemented to facilitate the functionality described herein. A representative hardware layer 804 is shown, and it may represent, for example, any of the computing devices described above. In some examples, the hardware layer 804 may be based on Fig. 9 It is implemented by the architecture of the computer system.

[0105] The representative hardware layer 804 includes one or more processing units 806 with associated executable instructions 808. The executable instructions 808 represent executable instructions of the software architecture 802, including implementations of the methods, modules, subsystems, components, etc. described herein, and may also include a memory and / or storage module 810 also having executable instructions 808. The hardware layer 804 may also include other hardware indicated by other hardware 812 and other hardware 822, which represent any other hardware of the hardware layer 804, such as other hardware shown as part of the software architecture 802.

[0106] exist Figure 8In the architecture of , software architecture 802 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, software architecture 802 may include layers such as operating system 814, library 816, framework / middleware layer 818, application 820, and presentation layer 844. In operation, application 820 or other components within a layer may call API call 824 through the software stack, and access responses, return values, etc. shown as messages 826 in response to API call 824. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware layer 818, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0107] The operating system 814 can manage hardware resources and provide public services. The operating system 814 can include, for example, a kernel 828, a service 830, and a driver 832. The kernel 828 can act as an abstraction layer between the hardware and other software layers. For example, the kernel 828 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The service 830 can provide other public services for other software layers. In some examples, the service 830 includes an interrupt service. The interrupt service can detect the reception of an interrupt, and in response, when the interrupt is accessed, the software architecture 802 is caused to suspend its current processing and execute an interrupt service routine (ISR).

[0108] The driver 832 may be responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the driver 832 may include a display driver, a camera driver, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, Near Field Communication (NFC) drivers, audio drivers, power management drivers, etc.

[0109] The library 816 can provide a common infrastructure that can be utilized by the application 820 or other components or layers. The library 816 generally provides functions that allow other software modules to perform tasks in a more convenient manner than directly interfacing with the underlying operating system 814 functions (e.g., kernel 828, service 830, or driver 832). The library 816 may include a system library 834 (e.g., C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 816 may include an API library 836, such as a media library (e.g., a library that supports the presentation and manipulation of multiple media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.), a graphics library (e.g., an OpenGL framework that can be used to present two-dimensional and three-dimensional graphics content on a display), a database library (e.g., SQLite that can provide multiple relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 816 may also include a variety of other libraries 838 to provide many other APIs to the application 820 and other software components / modules.

[0110] The framework / middleware layer 818 may provide a higher level common infrastructure that may be utilized by the applications 820 or other software components / modules. For example, the framework / middleware layer 818 may provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware layer 818 may provide various other APIs that may be used by the applications 820 or other software components / modules, some of which may be specific to a particular operating system or platform.

[0111] Applications 820 include built-in applications 840 or third-party applications 842. Representative examples of built-in applications 840 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, or game applications. Third-party applications 842 may include any built-in applications as well as a wide variety of other applications. In a specific example, third-party applications 842 (e.g., applications created by entities other than the vendor of a particular platform using Android TM or iOS TM Software Development Kit (SDK) can be used to develop applications that run on platforms such as iOS. TM 、Android TM , Windows Or mobile software on a mobile operating system of other mobile computing device operating systems. In this example, third-party applications 842 can call API calls 824 provided by a mobile operating system such as operating system 814 to facilitate the functions described herein.

[0112] Applications 820 may utilize built-in operating system functionality (e.g., kernel 828, services 830, or drivers 832), libraries (e.g., system libraries 834, API libraries 836, and other libraries 838), and framework / middleware layer 818 to create a user interface to interact with a user of the system. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 844. In these systems, the application / module "logic" may be separated from aspects of the application / module that interact with the user.

[0113] Some software architectures make use of virtual machines. Figure 8 In the example of , this is illustrated by virtual machine 848. A virtual machine creates a software environment in which applications / modules can execute as if they were executed on a hardware computing device. The virtual machine is hosted by a host operating system (operating system 814) and typically (although not always) has a virtual machine monitor 846 that manages the operation of the virtual machine and the interface with the host operating system (e.g., operating system 814). Software architecture is executed within virtual machine 848, such as operating system 850, library 852, framework / middleware 854, application 856, or presentation layer 858. These layers of the software architecture executed within virtual machine 848 can be the same as the corresponding layers previously described, or they can be different.

[0114] Certain examples are described herein as including logic or multiple components, modules, or mechanisms. A module may constitute a software module (e.g., code contained on (1) a non-transitory machine-readable medium or (2) a transmission signal) or a hardware-implemented module. A hardware-implemented module is a tangible unit that is capable of performing a specific operation and may be configured or arranged in a specific manner. In an example, one or more computer systems (e.g., stand-alone, client, or server computer systems) or one or more hardware processors may be configured by software (e.g., an application or application portion) as a hardware-implemented module that is used to perform certain operations described herein.

[0115] In various examples, hardware-implemented modules can be implemented mechanically or electronically. For example, a hardware-implemented module may include dedicated circuits or logic that are permanently configured (e.g., as a dedicated processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented module may also include programmable logic or circuits (e.g., contained in a general-purpose processor or another programmable processor) that are temporarily configured by software to perform certain operations. It should be understood that the decision to implement a hardware-implemented module mechanically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

[0116] Thus, the term "hardware-implemented module" should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily or provisionally configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. Considering an example in which a hardware-implemented module is temporarily configured (e.g., programmed), each hardware-implemented module need not be configured or instantiated at any one time. For example, where the hardware-implemented module includes a general-purpose processor configured using software, the general-purpose processor can be configured as different hardware-implemented modules at different times. The software can configure the processor accordingly, for example, to constitute a particular hardware-implemented module at one time, and to constitute different hardware-implemented modules at different times.

[0117] Hardware-implemented modules can provide information to other hardware-implemented modules and receive information from other hardware-implemented modules. Therefore, the described hardware-implemented modules can be considered to be communicatively coupled. In the case where multiple such hardware-implemented modules exist at the same time, communication can be achieved through signal transmission (for example, by connecting appropriate circuits and buses of the hardware-implemented modules). In examples where multiple hardware-implemented modules are configured or instantiated at different times, communication between such hardware-implemented modules can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware-implemented modules. For example, a hardware-implemented module can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware-implemented module can access the memory device at a later time to retrieve and process the stored output. Hardware-implemented modules can also initiate communication with input or output devices and can operate on resources (for example, information collections).

[0118] The various operations of the example methods described herein may be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the related operations. Whether temporarily or permanently configured, these processors may constitute processor-implemented modules for performing one or more operations or functions. In some examples, the modules mentioned herein may include processor-implemented modules.

[0119] Similarly, the method described herein can be implemented at least in part by a processor. For example, at least some operations of the method can be performed by one or more processors or a module implemented by the processor. The performance of some operations can be distributed in one or more processors, not only resident in a single machine, but also deployed on multiple machines. In some examples, one or more processors can be located in a single location (e.g., in a home environment, an office environment, or a server farm), and in other examples, the processor can be distributed in multiple locations.

[0120] The one or more processors may also operate to support performance of related operations in a "cloud computing" environment, or as "software as a service (SaaS)." For example, at least some operations may be performed by a group of computers (as an example of a machine including a processor), which may be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0121] The examples may be implemented in digital electronic circuitry, or in computer hardware, firmware, or software, or in combinations of these. The examples may be implemented using a computer program product, such as a computer program tangibly embodied in an information carrier, such as a computer program embodied in a machine-readable medium, for execution by, or to control the operation of, a data processing apparatus, such as a programmable processor, a computer, or multiple computers.

[0122] A computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, subroutine or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0123] In an example, the operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. The method operations may also be performed by a special purpose logic circuit (e.g., FPGA or ASIC), and some example devices may be implemented as a special purpose logic circuit.

[0124] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is due to the computer programs running on their respective computers, and they have a client-server relationship with each other. In the example of deploying a programmable computing system, it is understood that both hardware and software architectures are worth considering. Specifically, it should be understood that the choice of implementing a specific function in a combination of permanently configured hardware (e.g., ASIC), temporarily configured hardware (e.g., a combination of software and a programmable processor), or permanently and temporarily configured hardware can be a design choice. The following is the hardware (e.g., machine) and software architecture that can be deployed in a variety of examples.

[0125] Fig. 9It is a block diagram of a machine in the example form of a computer system 900, in which instructions 924 can be executed to make the machine perform any one or more methods discussed herein. In an optional example, the machine operates as a standalone device, or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can run as a server or client machine in a server-client network environment, or run as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a web device, a network router, a switch or a bridge, or any machine capable of executing instructions (sequential or other) specifying the action to be taken by the machine. In addition, although only a single machine is shown, the term "machine" should also be understood to include any machine collection that executes a set (or multiple sets) of instructions individually or jointly to perform any one or more methods discussed herein.

[0126] The example computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a GPU, or both), a main memory 904, and a static memory 906, which communicate with each other via a bus 908. The computer system 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 900 also includes an alphanumeric input device 912 (e.g., a keyboard or a touch-sensitive display screen), a UI navigation (or cursor control) device 914 (e.g., a mouse), a storage unit 916, a signal generating device 918 (e.g., a speaker), and a network interface device 920.

[0127] The storage unit 916 includes a machine-readable medium 922 on which is stored one or more sets of data structures and instructions 924 (e.g., software) that embody or are utilized by any one or more of the methodologies or functions described herein. During execution of the instructions 924 by the computer system 900, the instructions 924 may also reside in whole or in part in the main memory 904 or in the processor 902, each of which also constitutes a machine-readable medium 922.

[0128] Although according to some examples, the machine-readable medium 922 is shown as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) storing one or more instructions 924 or data structures. The term "machine-readable medium" should also be understood to include any tangible medium that can store, encode, or carry instructions 924 executed by a machine and cause the machine to perform any one or more methods of the present disclosure, or any tangible medium that can store, encode, or carry data structures used by or associated with such instructions 924. Therefore, the term "machine-readable medium" should include, but is not limited to, solid-state memory, optical and magnetic media. Specific examples of machine-readable media 922 include non-volatile memory, including, as examples, semiconductor storage devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and compact disk read-only memory (CD-ROM) and digital versatile disk read-only memory (DVD-ROM) disks. Machine-readable media are not transmission media.

[0129] The instructions 924 may also be sent or received over a communication network 926 using a transmission medium. The instructions 924 may be transmitted using a network interface device 920 and any of a number of well-known transmission protocols, such as the Hypertext Transfer Protocol (HTTP). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, a mobile phone network, a plain old telephone (POTS) network, and wireless data networks (e.g., Wi-Fi and Wi-Max networks). The term "transmission medium" shall be understood to include any intangible medium capable of storing, encoding, or carrying the instructions 924 to be executed by the machine, and includes digital or analog communication signals or other intangible media to facilitate communication of such software.

[0130] Although specific examples are described herein, it is apparent that various modifications and changes may be made to these examples without departing from the broader spirit and scope of the present disclosure. Therefore, the specification and the drawings are to be considered illustrative rather than restrictive. The drawings forming part of this article illustrate specific examples in which the present subject matter may be practiced by way of illustration and not limitation. The illustrated examples are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other examples may be utilized and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, this detailed description should not be construed as limiting, and the scope of the various examples is defined only by the appended claims and the full scope of equivalents authorized by these claims.

[0131] These examples of the subject matter of the present invention may be referred to herein individually or collectively as "examples", which is merely for convenience, and if more than one example or concept is actually disclosed, this does not mean that the scope of the present application is voluntarily limited to any single example or concept. Therefore, although specific examples have been shown and described herein, it should be understood that any arrangement intended to achieve the same purpose can replace the specific examples shown. The present disclosure is intended to cover any and all modifications or variations of the various examples. By reading the above description, those skilled in the art will understand the combination of the above examples and other examples not specifically described herein.

[0132] Some parts of the subject matter discussed herein can be presented according to an algorithm or symbolic representation of the operation of data stored in a machine memory (e.g., computer memory) as a bit or binary digital signal. Such an algorithm or symbolic representation is an example of a technique used by a person of ordinary skill in the field of data processing to convey the essence of their work to other persons of ordinary skill in the art. As used herein, an "algorithm" is a self-consistent sequence of operations or similar processes that result in a desired result. In this case, algorithms and operations involve physical operations of physical quantities. Typically, but not necessarily, these quantities can take the form of electrical, magnetic or optical signals that can be stored, accessed, transmitted, combined, compared or otherwise manipulated by a machine. Mainly for general reasons, it is sometimes convenient to use words such as "data", "content", "bit", "value", "element", "symbol", "character", "term", "digit", "quantity symbol" to refer to such a signal. However, these words are merely convenient tags and will be associated with appropriate physical quantities.

[0133] Unless otherwise stated, discussions herein using words such as "processing," "computing," "calculating," "determining," "performing," "displaying," and the like may refer to the action or process of a machine (e.g., a computer) that manipulates or transforms data representing physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components to receive, store, send, or display information. In addition, unless otherwise stated, the terms "a" and "an" are used herein to include one or more than one instance, as is common in patent literature. Finally, as used herein, the conjunction "or" refers to a non-exclusive "or" unless expressly stated otherwise.

[0134] Unless the context clearly requires otherwise, throughout the specification and claims, the words "include", "comprising", etc. should be interpreted as inclusive, rather than exclusive or exhaustive, such as "including, but not limited to". As used herein, the terms "connected", "coupled" or any variation thereof mean any direct or indirect connection or coupling between two or more elements; the coupling or connection between elements may be physical, logical, or a combination thereof. In addition, when used in this application, the words "herein", "above", "below" and words of similar meaning refer to the entirety of this application, not to any particular part of this application. Where the context permits, words using the singular or plural may also include the plural or singular, respectively. The word "or" referring to a list of two or more items covers all of the following interpretations of the word: any one item in the list, all items in the list, and any combination of items in the list.

[0135] Although some examples, such as the examples shown in the accompanying drawings, include a specific order of operations, this order may be changed without departing from the scope of the present disclosure. For example, some of the operations described may be performed in parallel or in a different order, which will not substantially affect the functions described in the examples. In other examples, different components of the example devices or systems that implement the example methods may perform functions substantially simultaneously or in a specific order.

Claims

1. A system comprising: at least one memory storing instructions; as well as One or more processors configured by the instructions to perform operations including: generating an inverted index including one or more restricted emission lists based on the source data and the emission list threshold, each restricted emission list having a maximum size corresponding to the emission list threshold, and the source data including a plurality of records; receiving a search query including a value identifying a restricted emissions list of the one or more restricted emissions lists; in response to receiving the search query, retrieving one or more record identifiers from the identified restricted emissions list using the value, each record identifier uniquely identifying a record from the plurality of records; and The one or more record identifiers are returned.

2. The system according to claim 1, wherein: Generating an inverted index involves, for each restricted emission list: populating a restricted emissions list with at least one record identifier; generating a flag indicating whether an emissions list threshold has been reached with respect to a restricted emissions list; as well as Stores a flag associated with a restricted emissions list.

3. The system of claim 2, the operations further comprising, in response to receiving a search query: detecting a need for a fallback search based at least in part on the flag; and In response to detecting that a fallback search is needed, a fallback search is performed using the value to retrieve from the source data at least one record identifier that could not be retrieved from the identified restricted emissions list.

4. The system of claim 3, wherein performing a fallback search comprises: Identify the range or subset of records in the source data that are not covered by the identified restricted emissions list; as well as Searches a range or subset of records to identify records that contain the value.

5. The system according to claim 1, wherein: Generating the inverted index includes populating each restricted emission list with record identifiers of records in the source data until an emission list threshold is reached.

6. The system according to claim 5, wherein: Each restricted emissions list is populated based on a predetermined order.

7. The system according to claim 6, wherein: The predetermined order is a reverse chronological order.

8. The system according to claim 1, wherein: The inverted index comprises a plurality of emission lists, and wherein the emission list threshold is a global emission list threshold applied to each emission list of the inverted index such that each emission list is a restricted emission list.

9. The system according to claim 1, wherein: The inverted index comprises a plurality of emission lists, wherein the source data comprises values ​​of a plurality of attributes, and wherein the emission list threshold is an attribute-specific emission list threshold applied to the one or more restricted emission lists, the one or more restricted emission lists being a subset of the plurality of emission lists.

10. The system according to claim 1, wherein: The inverted index comprises a plurality of emission lists, wherein the source data comprises a plurality of tables, and wherein the emission list threshold is a table-specific emission list threshold applied to the one or more restricted emission lists, the one or more restricted emission lists being a subset of the plurality of emission lists.

11. The system of claim 1, the operations further comprising: User input is received to select an emissions list threshold.

12. The system of claim 1, the operations further comprising: Analyze records in source data; as well as Select the emission listing threshold based on analyzing the records in the source data.

13. The system according to claim 1, wherein the operation further comprises, after generating the inverted index: receiving user input to adjust the emissions list threshold; and In response to receiving user input to adjust the emissions list threshold: Define adjusted emissions inventory thresholds, and The maximum size of each restricted emission list is adjusted to correspond to the adjusted emission list threshold.

14. The system of claim 1, wherein: The source data includes at least one table, wherein the records are represented as corresponding rows in the at least one table, and each table further includes at least one column, each column represents an attribute, wherein unique values ​​of the attributes are used as corresponding keys in the inverted index.

15. A method comprising: generating an inverted index including one or more restricted emission lists based on the source data and the emission list threshold, each restricted emission list having a maximum size corresponding to the emission list threshold, and the source data including a plurality of records; receiving a search query including a value identifying a restricted emissions list of the one or more restricted emissions lists; in response to receiving the search query, retrieving one or more record identifiers from the identified restricted emissions list using the value, each record identifier uniquely identifying a record from the plurality of records; and The one or more record identifiers are returned.

16. The method according to claim 15, wherein: Generating an inverted index involves, for each restricted emission list: populating a restricted emissions list with at least one record identifier; generating a flag indicating whether an emissions list threshold has been reached with respect to a restricted emissions list; as well as Stores a flag associated with a restricted emissions list.

17. The method of claim 16, further comprising, in response to receiving a search query: detecting a need for a fallback search based at least in part on the flag; and In response to detecting that a fallback search is needed, a fallback search is performed using the value to retrieve from the source data at least one record identifier that could not be retrieved from the identified restricted emissions list.

18. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: generating an inverted index including one or more restricted emission lists based on the source data and the emission list threshold, each restricted emission list having a maximum size corresponding to the emission list threshold, and the source data including a plurality of records; receiving a search query including a value identifying a restricted emissions list of the one or more restricted emissions lists; in response to receiving the search query, retrieving one or more record identifiers from the identified restricted emissions list using the value, each record identifier uniquely identifying a record from the plurality of records; and The one or more record identifiers are returned.

19. The non-transitory computer readable medium of claim 18, wherein: Generating an inverted index involves, for each restricted emission list: populating a restricted emissions list with at least one record identifier; generating a flag indicating whether an emissions list threshold has been reached with respect to a restricted emissions list; as well as Stores a flag associated with a restricted emissions list.

20. The non-transitory computer-readable medium of claim 19, the operations further comprising, in response to receiving a search query: detecting a need for a fallback search based at least in part on the flag; and In response to detecting that a fallback search is needed, a fallback search is performed using the value to retrieve from the source data at least one record identifier that could not be retrieved from the identified restricted emissions list.

Images