Cached updateable TOP-K index

Abstract

A method is provided that stores an index structure in a secondary memory, the index structure including, for each given word from among words included in a document, a group of document IDs of documents containing the given word. The method stores a subset of the index structure in a primary memory that is faster than the secondary memory. The method obtains a keyword and identifies any document that includes the keyword. The method finds the top-K frequent words among the words included in the identified documents in the following manner: for each given group listed in descending order by the number of document IDs therein, identifying the number of document IDs of the documents identified in the given group from the subset when the number of document IDs in the given group is within a range, otherwise identifying from the index structure; and presenting the words of the top-K groups having the largest number of identified document IDs.

Description

Background Art

[0001] The present invention relates generally to data storage and, more particularly, to an updateable top-k index for a cache.

[0002] In the field of text analytics, finding the top-k frequent words (top-k for short) in a collection of documents, typically given dynamically by, for example, document search, has proven essential in finding new insights from data.

[0003] Several techniques exist for efficiently computing the top-k frequent words for a given search query. However, in these techniques, the index structure is difficult to update and is not suitable for use cases where new documents frequently enter. Furthermore, in these techniques, the index may be suitable for updating, but suffers from query runtime issues due to the need for extensive random access to secondary storage devices (e.g., hard disk drives (HDDs)). Summary of the Invention

[0004] According to aspects of the present invention, a computer-implemented method for storing data is provided. The method includes storing an index structure in a secondary memory. The index structure includes, for each given word among the words included in the document, a group of document IDs of documents containing the given word. The groups in the index structure are listed in descending order by the number of document IDs therein. The method also includes storing a subset of the index structure in a main memory having a faster access time than the secondary memory. The number of document IDs in each of the groups in the subset is N a and N b In the range between a >N b . The method also includes obtaining a keyword from the query. The method also includes identifying any document that includes the keyword. The method also includes finding the top-K frequent words among the words included in the identified documents by the following steps: for each given group in the groups arranged in descending order of the number of document IDs in the given group, identifying the number of identified document IDs in the given group, when the number of document IDs in the given group is within a range, identifying the number of document IDs of the identified documents in the given group from the subset in the main memory, and when the number of document IDs in the given group is not within the range, identifying the number of document IDs of the identified documents in the given group from the index structure in the auxiliary memory; and presenting the words of the top-K groups with the largest number of identified document IDs.

[0005] According to another aspect of the present invention, a computer program product for storing data is provided. The computer program product includes a non-transitory computer-readable storage medium having program instructions embodied therein. The program instructions are executable by a computer to cause the computer to perform a method. The method includes storing an index structure in an auxiliary memory. The index structure includes, for each given word among the words included in the document, a group of document IDs of documents containing the given word. The groups in the index structure are listed in descending order by the number of document IDs therein. The method also includes storing a subset of the index structure in a main memory having a faster access time than the auxiliary memory. The number of document IDs in each of the groups in the subset is N a and N b In the range between a >N b . The method also includes obtaining a keyword from the query. The method also includes identifying any document that includes the keyword. The method also includes finding the top-K frequent words among the words included in the identified documents by the following steps: for each given group in the groups arranged in descending order of the number of document IDs in the given group, identifying the number of identified document IDs in the given group, when the number of document IDs in the given group is within a range, identifying the number of document IDs of the identified documents in the given group from the subset in the main memory, and when the number of document IDs in the given group is not within the range, identifying the number of document IDs of the identified documents in the given group from the index structure in the auxiliary memory; and presenting the words of the top-K groups with the largest number of identified document IDs.

[0006] According to another aspect of the present invention, a computer processing system for determining a correspondence between a command and a process is provided. The computer processing system includes a memory device including program code stored thereon. The computer processing system also includes a hardware processor operatively coupled to the memory device and configured to execute the program code stored on the memory device to store a subset of the index structure in a main memory having a faster access time than the auxiliary memory. The number of document IDs in each of the groups in the subset is N a and N b In the range between a >N b. The hardware processor is also configured to run the program code to obtain keywords from the query. The hardware processor is also configured to run the program code to identify any one of the documents that include the keywords. The hardware processor is further configured to find the top-K frequent words among the words included in the identified documents by the following operations: for each given group in the groups arranged in descending order of the number of document IDs in the given group, identifying the number of identified document IDs in the given group, when the number of document IDs in the given group is within a range, identifying the number of document IDs of the identified documents in the given group from the subset in the main memory, and when the number of document IDs in the given group is not within the range, identifying the number of document IDs of the identified documents in the given group from the index structure in the auxiliary memory; and presenting the words of the top-K groups with the largest number of identified document IDs.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments of the invention, which is to be read in connection with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The following description will provide details of the preferred embodiments with reference to the following drawings, in which:

[0009] Figure 1 is a block diagram illustrating an exemplary processing system according to an embodiment of the present invention;

[0010] Figure 2 is a block diagram illustrating an exemplary environment in which the present invention may be applied according to an embodiment of the present invention;

[0011] Figure 3 is a flow chart illustrating an exemplary method for data storage according to an embodiment of the present invention;

[0012] Figure 4 is a flowchart illustrating another exemplary method for data storage according to an embodiment of the present invention;

[0013] Figure 5 is a plot of a graph of the top-K index of a cache according to an embodiment of the present invention;

[0014] Figure 6 is a block diagram showing an illustrative cloud computing environment having one or more cloud computing nodes with which local computing devices used by cloud consumers communicate, in accordance with an embodiment of the present invention; and

[0015] Figure 7 is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Embodiments of the present invention relate to an updateable top-k index of a cache.

[0017] One or more embodiments of the present invention utilize a memory cache that enables efficient update and run times with a limited amount of main memory.

[0018] One or more embodiments of the present invention cache infrequent words in a main memory (e.g., random access memory (RAM)) to minimize random access to a secondary device (e.g., a hard disk drive (HDD)), as opposed to the standard use of a cache to store the most frequent objects that are most likely to be accessed. Here, the main memory essentially operates as a cache for the hard disk drive, storing a subset of an index. The subset stored in the main memory includes words with a moderate frequency of occurrence, while the secondary device stores the remainder of the index, including words with the highest (and optionally the lowest) frequency of occurrence.

[0019] Figure 1 1 is a block diagram illustrating an exemplary processing system 100 according to an embodiment of the present invention. Processing system 100 includes a set of processing units (e.g., CPUs) 101, a set of GPUs 102, a set of memory devices 103, a set of communication devices 104, and a set of peripheral devices 105. CPU 101 can be a single-core or multi-core CPU. GPU 102 can be a single-core or multi-core GPU. One or more memory devices 103 can include cache, RAM, ROM, and other memories (flash memory, optical memory, magnetic memory, etc.). Communication device 104 can include wireless and / or wired communication devices (e.g., network (e.g., WIFI, etc.) adapters, etc.). Peripheral devices 105 can include display devices, user input devices, printers, imaging devices, etc. The elements of processing system 100 are connected via one or more buses or networks (collectively represented by reference numeral 110).

[0020] In an embodiment, the memory device 103 may store specially programmed software modules to convert a computer processing system into a special-purpose computer configured to implement various aspects of the present invention. In an embodiment, dedicated hardware (e.g., an application-specific integrated circuit, a field programmable gate array (FPGA), etc.) may be used to implement various aspects of the present invention. In an embodiment, the memory device 103 includes a main memory 103A and a hard disk drive (HDD) 103B coupled to a memory controller 103C. The memory controller 103C is configured to control the storage of data to the main memory 103A and the HDD 103B. In an embodiment, the memory controller 103C may be at least partially implemented in hardware by the CPU 101 or the GPU 102.

[0021] Of course, as will be readily apparent to those skilled in the art, the processing system 100 may also include other elements (not shown), as well as omit certain elements. For example, as will be readily apparent to those skilled in the art, various other input devices and / or output devices may be included in the processing system 100 depending on their specific implementation. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memories, etc. in various configurations may also be utilized. Furthermore, in another embodiment, a cloud configuration (e.g., see Figure 6-7 These and other variations on processing system 100 will be apparent to those of ordinary skill in the art given the teachings of the present invention provided herein.

[0022] Furthermore, it should be understood that the various figures described below with respect to various elements and steps related to the present invention may be implemented in whole or in part by one or more elements of system 100 .

[0023] As used herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory, software, or a combination thereof that collaborate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.). In some embodiments, the hardware processor subsystem may include one or more memories, which may be on-board or off-board, or may be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0024] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0025] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0026] These and other variations in hardware processor subsystems are also contemplated according to embodiments of the present invention.

[0027] Figure 2is a block diagram illustrating an exemplary environment 200 in which the present invention may be applied, according to an embodiment of the present invention.

[0028] The environment 200 includes a remote server 210 and a group of user devices 220. The group of user devices 220 can access the remote server 210 through one or more networks 230. Each of the group of user devices 220 can be similar to Figure 1 The computer processing system 100 may be configured, or may be implemented by different devices (eg, a mobile phone, a tablet computer, etc.).

[0029] The remote server 210 includes a document repository 210A from which the user device 220 searches and retrieves documents for local storage. Local storage is implemented by the user device 220 in an efficient manner so as to minimize access to secondary memory (cache). In an embodiment, the removal server 210 may be implemented using a cloud configuration (see, e.g., Figure 6-7 For example, the document may be provided as a cloud service for document search or document retrieval. In another embodiment, the remote server 220 may be configured as an infrastructure as a service, where the storage using the present invention is executed on the server and accessed by software on the user device 220. For example, the user device may include a document search engine that retrieves documents based on entered keywords, where documents and / or top-k words are provided to the user based on the entered keywords.

[0030] Figure 3 is a flow chart illustrating another exemplary method 300 for data storage according to an embodiment of the present invention.

[0031] At block 305, an index structure is stored in the auxiliary memory, the index structure including, for each given word among the words included in the document, a group of document IDs of documents containing the given word. The groups in the index structure are listed in descending order by the number of document IDs therein.

[0032] At block 310, a subset of the index structure is stored in primary memory, which has a faster access time than secondary memory. The number of document IDs in each of the groups in the subset is N a and N b In the range between a >N b .

[0033] At block 315 , keywords are obtained from the query.

[0034] At block 320 , any documents that include the keyword are identified.

[0035] At block 325 , the top-K frequent words are found among the words included in the identified document.

[0036] In an embodiment, block 325 may include blocks 325A and 325B.

[0037] In box 325A, for each given group in the groups arranged in descending order of the number of document IDs in the group, identify the number of identified document IDs in the given group, when the number of document IDs in the given group is within a range, identify from the subset in the main memory, and when the number of document IDs in the given group is not within the range, identify from the index structure in the secondary memory.

[0038] At block 325B, the top-K groups of words with the largest number of identified document IDs are presented.

[0039] Figure 4 is a flow chart illustrating yet another exemplary method 400 for data storage according to an embodiment of the present invention.

[0040] The method 400 operates according to the following: a maximum value of the size of the cache is given, for example M (eg, 10 megabytes).

[0041] Method 400 involves an index building section 491 and a query processing section 492. The input to index building section 491 is a collection of documents. The output of index building section 491 is an index on auxiliary devices. The input to query processing section 492 is D, a list of document IDs, which serves as a query. The output of query processing section 492 is the top-K words with their frequencies.

[0042] In block 405, an updateable index structure Idx is constructed on the secondary device, which returns a list of document IDs given any keyword using existing techniques. n Is freq(w1)≥…≥freq(w n ) for all words. i ) is w i The number of documents.

[0043] At block 410, before query processing, a cache C of size at most M is established in memory such that the cache holds w a 、w a+1 、…、w b (a<b) index structure, where a and b are predefined numbers, so that for w a 、w a+1 、…、w b A subset of the index of can be stored in a cache of size M, where M is defined as the maximum number of document IDs. For example, if b = n and a is used, then the condition is satisfied such that M ≥ freq(wa )+freq(w a+1 )+…+freq(w b ).

[0044] At block 415, we start by taking the empty set S as the container for the top-K words. For each i=1, 2, ..., n, we do the following:

[0045] If a≤i≤b, then D i , w from C i Otherwise, get D from Idx i ;as well as

[0046] If |S|=K and [the frequency of the least frequent word in K]≥|D i |, then return S; otherwise, pass (w i ,|D∩D i |) Update S. The value of i in this step is called early termination, and it can also be used as an estimate of the value of b after query processing is complete. This estimate can be used for future queries. Another use of early output is to collect the early output as a result of processing multiple queries and take b as their average.

[0047] A description will now be given on variations for obtaining a good range (a, b) for cache according to an embodiment of the present invention.

[0048] Calculate the query processing time for various (random) queries with different frequencies, and then estimate the value of (a, b) that satisfies the conditions of memory usage and the expected upper bound in processing time.

[0049] Figure 5 is a plot of a graph of a cached top-K index 500 according to an embodiment of the present invention.

[0050] As can be seen, there is an area 520 of the main memory (eg, RAM) cached as part of the present invention that includes words with a medium (intermediate) frequency of occurrence. In contrast, a typical cache includes words with a maximum access frequency 510.

[0051] It should be understood that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings set forth herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0052] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0053] Features are as follows:

[0054] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed without manual interaction with the service provider.

[0055] Wide Area Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0056] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. This is location-independent in the sense that consumers typically do not control or know the exact location of the provided resources, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0057] Rapid elasticity: In some cases, the ability to scale out quickly and in quickly can be provided quickly and elastically. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0058] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the provider and consumer of the utilized service.

[0059] The service model is as follows:

[0060] Software as a Service (SaaS): The ability provided to consumers is to use the provider's applications running on a cloud infrastructure. Applications are accessed from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0061] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications onto cloud infrastructure. These applications are built using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but do have control over the deployed applications and possibly the configuration of the application hosting environment.

[0062] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0063] The deployment model is as follows:

[0064] Private cloud: The cloud infrastructure is operated solely for the organization. It can be managed by the organization or a third party and can exist inside or outside the building.

[0065] Community cloud: Cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0066] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0067] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a unique entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0068] The cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure consisting of a network of interconnected nodes.

[0069] Now refer to Figure 6, depicts an illustrative cloud computing environment 650. As shown, the cloud computing environment 650 includes one or more cloud computing nodes 610 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 654A, a desktop computer 654B, a laptop computer 654C, and / or an automobile computer system 654N. The nodes 610 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 650 to provide infrastructure, platform, and / or software as a service for which the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Figure 6 The types of computing devices 654A-N shown in FIG are intended to be illustrative only, and computing node 610 and cloud computing environment 650 may communicate with any type of computerized device over any type of network and / or network-addressable connection (eg, using a web browser).

[0070] Now refer to Figure 7 , showing the cloud computing environment 650 ( Figure 6 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 7 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0071] The hardware and software layer 760 includes hardware and software components. Examples of hardware components include: host 761; server 762 based on RISC (Reduced Instruction Set Computer) architecture; server 763; blade server 764; storage device 765; and network and networking components 766. In some embodiments, software components include network application server software 767 and database software 768.

[0072] Virtualization layer 770 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 771 ; virtual storage 772 ; virtual networks 773 , including virtual private networks; virtual applications and operating systems 774 ; and virtual clients 775 .

[0073] In one example, the management layer 780 may provide the functionality described below. Resource provisioning 781 provides dynamic procurement of computing resources and other resources used to perform tasks within a cloud computing environment. Metering and pricing 782 provides cost tracking when utilizing resources in a cloud computing environment, as well as accounting or invoicing for the consumption of those resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 783 provides access to the cloud computing environment for consumers and system administrators. Service level management 784 provides cloud computing resource allocation and management so that required service levels are met. Service level agreement (SLA) planning and fulfillment 785 provides pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0074] The workload layer 790 provides examples of functionality that a cloud computing environment can be used for. Examples of workloads and functionality that can be provided from this layer include: mapping and navigation 791; software development and lifecycle management 792; virtual classroom education delivery 793; data analytics processing 794; transaction processing 795; and cached, updateable top-k indexes 796.

[0075] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0076] A computer-readable storage medium can be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0077] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0078] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0079] Aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowcharts and / or block diagrams and the combination of blocks in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing device and / or other equipment to operate in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0082] References in the specification to "one embodiment" or "an embodiment" of the present invention, and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment, is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment," and any other variations thereof, in various places throughout the specification are not necessarily all referring to the same embodiment. However, it should be understood that features of one or more embodiments may be combined, given the teachings of the present invention provided herein.

[0083] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to as many items as are listed.

[0084] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing the specified logical function. In some alternative embodiments, the functions noted in the frame may not occur in the order noted in the figure. For example, the two frames shown in succession can actually be implemented as a step, simultaneously, substantially simultaneously, in a manner that overlaps part or all of the time, or these frames can sometimes be performed in reverse order, depending on the functions involved. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0085] Having described preferred embodiments of the systems and methods (which are intended to be illustrative and not limiting), it is noted that modifications and variations may be made by those skilled in the art in light of the above teachings. Therefore, it should be understood that changes may be made in the particular embodiments disclosed that are within the scope of the invention as outlined by the appended claims. Having thus described the aspects of the present invention with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.

Claims

1. A computer-implemented method for storing data, comprising: storing an index structure in an auxiliary memory, the index structure including, for each given word from among words included in a document, a group of document IDs of documents containing the given word, wherein the groups in the index structure are listed in descending order of the number of document IDs therein; A subset of the index structure is stored in a main memory having a faster access time than the auxiliary memory, the number of document IDs in each of the groups in the subset being N a and N b In the range between a >N b ; Get keywords from the query; identifying any documents among the documents that include the keyword; Find the top-K frequent words among the words included in the identified documents by the following steps: for each given group of the groups arranged in descending order of the number of document IDs in the given group, identifying the number of document IDs of the identified documents in the given group, identifying the number of document IDs of the identified documents in the given group from the subset in the primary memory when the number of document IDs in the given group is within the range, and identifying the number of document IDs of the identified documents in the given group from the index structure in the secondary memory when the number of document IDs in the given group is not within the range; as well as The top-K groups of words with the largest number of identified document IDs are presented.

2. The computer-implemented method of claim 1 , wherein: The main memory is a random access memory RAM, and the auxiliary memory is a hard disk drive HDD.

3. The computer-implemented method of claim 1 , wherein: The index structure is configured to be updateable.

4. The computer-implemented method of claim 1 , further comprising selectively updating the index structure or a subset of the index structure by adding a new group to the index structure or a subset of the index structure in response to a request to add a new document to the index structure.

5. The computer-implemented method of claim 1 , wherein: For each corresponding group in the group, N b A value that is set to the number of words in the corresponding one of the documents.

6. The computer-implemented method of claim 1 , wherein: For each corresponding group in the set, N is estimated by averaging the early outputs of past queries. b .

7. The computer-implemented method of claim 1 , wherein: The groups in the subset exclude the corresponding one of the groups having the maximum number of document IDs therein.

8. The computer-implemented method of claim 1 , further comprising: a The value of is preset to be greater than the number of occurrences of the most frequently occurring word among the words in the document.

9. The computer-implemented method of claim 1 , further comprising estimating N that satisfies the conditions of memory usage and the upper bound of expected processing time. a and N b value.

10. The computer-implemented method of claim 1 , wherein: The range is selected to include words having a medium access occurrence frequency relative to the remaining ones of the words.

11. A computer program product for storing data, the computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions being executable by a computer to cause the computer to perform a method comprising: storing an index structure in an auxiliary memory, the index structure including, for each given word from among words included in a document, a group of document IDs of documents containing the given word, wherein the groups in the index structure are listed in descending order of the number of document IDs therein; A subset of the index structure is stored in a main memory having a faster access time than the auxiliary memory, the number of document IDs in each of the groups in the subset being N a and N b In the range between a >N b ; Get keywords from the query; identifying any documents among the documents that include the keyword; Find the top-K frequent words among the words included in the identified documents by the following steps: for each given group of the groups arranged in descending order of the number of document IDs in the given group, identifying the number of document IDs of the identified documents in the given group, identifying the number of document IDs of the identified documents in the given group from the subset in the primary memory when the number of document IDs in the given group is within the range, and identifying the number of document IDs of the identified documents in the given group from the index structure in the secondary memory when the number of document IDs in the given group is not within the range; as well as The top-K groups of words with the largest number of identified document IDs are presented.

12. The computer program product of claim 11, wherein: The main memory is a random access memory RAM, and the auxiliary memory is a hard disk drive HDD.

13. The computer program product of claim 11, wherein the index structure is configured to be updateable.

14. The computer program product of claim 11, further comprising selectively updating the index structure or the subset of the index structure by adding a new group to the index structure or the subset of the index structure in response to a request to add a new document to the index structure.

15. The computer program product of claim 11, wherein: For each corresponding group in the group, N b A value that is set to the number of words in the corresponding one of the documents.

16. The computer program product of claim 11, wherein: For each corresponding group in the set, N is estimated by averaging the early outputs of past queries. b .

17. The computer program product of claim 11, wherein: The groups in the subset exclude the corresponding one of the groups having the maximum number of document IDs therein.

18. The computer program product of claim 11, further comprising: a The value of is preset to be greater than the number of occurrences of the most frequently occurring word among the words in the document.

19. The computer program product of claim 11, further comprising estimating N that satisfies the conditions of memory usage and the upper limit of expected processing time. a and N b value.

20. A computer processing system for determining command-to-process correspondence, comprising: a memory device including program code stored thereon; A hardware processor operatively coupled to the memory device and configured to execute program code stored on the memory device to A subset of the index structure is stored in the main memory having a faster access time than the auxiliary memory, wherein the number of document IDs in each of the groups is N a and N b In the range between a >N b ; Get keywords from the query; identifying any of the documents that include the keyword; Find the top-K frequent words among the words included in the identified documents by the following steps: for each given group of the groups arranged in descending order of the number of document IDs in the given group, identifying the number of document IDs of the identified documents in the given group, and when the number of document IDs in the given group is within the range, identifying the number of document IDs of the identified documents in the given group from the subset in the main memory, and When the number of document IDs in the given group is not within the range, identifying the number of document IDs of the identified documents in the given group from the index structure in the auxiliary memory; as well as The top-K groups of words with the largest number of identified document IDs are presented.

Images