Sparse data index table

Abstract

The present invention relates to a sparse data index table. A set of queries for a set of data records is received. A set of sub-conditions required to support the set of queries is determined. An index table is maintained, wherein entries in the index table indicate sub-condition states associated with data records in the set of data records. A specific query indicating a specified set of sub-condition states is received. The specified set of sub-condition states is matched with the index table. A result set corresponding to matching rows in the index table is returned, wherein the matching rows match the specified set of sub-condition states.

Description

[0001] Cross-references to other applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 789,410, filed on January 7, 2019, entitled SPARSE DATA INDEX TABLE, and U.S. Provisional Patent Application No. 62 / 799,609, filed on January 31, 2019, entitled INDEX TABLE AS QUERY SUPPORT ON SPARSE DATA, both of which are incorporated herein by reference for all purposes. Technical Field

[0003] The present invention relates to the field of databases, and more particularly to sparse data index tables. Background Art

[0004] Databases organize data for querying. Indexing is a technique used to improve database query performance. In some cases, traditional indexing may be inefficient, creating space overhead without significantly reducing query costs. Therefore, there is a need to improve indexing for databases for such situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0006] Figure 1 is a functional diagram illustrating a programmed computer / server system for sparse data indexing tables according to some embodiments.

[0007] Figure 2 is an illustration of an embodiment of a conventional bitmap index that logically references certain rows in a base data table.

[0008] Figure 3 is a diagram of an index table.

[0009] Figure 4A is a graphic representation of an index row mask.

[0010] Figure 4B is an embodiment of a process of matching operation.

[0011] Figure 5A Is an illustration of a failure scenario vector that is a symptom.

[0012] Figure 5B is a graphic representation of a root cause table (RCT).

[0013] Figure 5Cis a diagram of a 64-bit block representation of known and value bits.

[0014] Figure 5D is a graphic representation of the root cause analysis technique.

[0015] Figure 5E It is a diagram of the RCT hierarchy.

[0016] Figure 6 is a diagram of the query flow using an index table.

[0017] Figure 7 is a diagram of a preferred layout of an index table.

[0018] Figure 8 is a flow chart illustrating an embodiment of a process of querying data records using an index table. DETAILED DESCRIPTION

[0019] The present invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on a memory coupled to the processor and / or provided by the memory. In this specification, these implementations, or any other form that the present invention may take, may be referred to as techniques. In general, the order of the steps of the disclosed process may be changed within the scope of the present invention. Unless otherwise stated, components such as processors or memories described as being configured to perform a task may be implemented as general components that are temporarily configured to perform the task at a given time or as specific components that are manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0020] The following together with the attached drawings illustrating the principles of the invention Figure 1 A detailed description of one or more embodiments of the invention is provided together. The invention is described in conjunction with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims, and the invention encompasses numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material known in the technical field related to the invention is not described in detail so as not to unnecessarily obscure the invention.

[0021] Efficient indexing of sparse data sets is disclosed. As referred to herein, a table can be sparse in the sense that there are a large number of entries containing what is known in database terminology as a NULL value. The NULL value typically corresponds to "not known" or "don't care." One embodiment is an entry corresponding to an "unknown" value, which corresponds to an entry value that is not known. Another embodiment is an entry corresponding to a "don't care" value, which is also called an insignificant value, which corresponds to an entry that is not needed for the particular analysis being used.

[0022] For applications to sparse data tables, use of an index table having at least one bitmap index and extended with the ability to indicate a "don't care" value is disclosed. The index table for a given row includes sub-condition values ​​corresponding to index sub-conditions of the data table. A matching mechanism is disclosed in which an index table and an index row mask are input to return a list of rows matching the mask using ternary and / or quaternary matching.

[0023] Figure 1 is a functional diagram illustrating a programmed computer / server system for a sparse data index table according to some embodiments. As shown, Figure 1 A functional illustration of a general purpose computer system programmed to associate with a sparse data index table is provided according to some embodiments. As will be apparent, other computer system architectures and configurations may be used for a sparse data index table.

[0024] The computer system 100, including various subsystems described below, includes at least one microprocessor subsystem, also referred to as a processor or central processing unit ("CPU") (102). For example, the processor (102) can be implemented by a single-chip processor or by multiple cores and / or processors. In some embodiments, the processor (102) is a general-purpose digital processor that controls the operation of the computer system 100. Using instructions retrieved from the memory (110), the processor (102) controls the receipt and manipulation of input data, as well as the output and display of data on output devices, such as a display and a graphics processing unit (GPU) (118).

[0025] The processor (102) is bidirectionally coupled to a memory (110), which may include a first main storage device, typically a random access memory ("RAM"), and a second main storage area, typically a read-only memory ("ROM"). As is known in the art, the main storage device can be used as a general storage area and as a scratch-pad memory, and can also be used to store input data and processed data. The main storage device can also store programming instructions, as well as data in the form of data objects and text objects, in addition to other data and instructions for process operations on the processor (102). As is also known in the art, the main storage device typically includes basic operating instructions, program code, data, and objects used by the processor (102) to perform its functions, such as programmed instructions. For example, the main storage device (110) may include any suitable computer-readable storage medium described below, depending on, for example, whether data access needs to be bidirectional or unidirectional. For example, the processor (102) can also directly and very quickly retrieve and store frequently needed data in a cache memory not shown. The processor (102) may also include co-processors (not shown) as supplemental processing components to assist the processor and / or the memory (110).

[0026] The removable mass storage device (112) provides additional data storage capacity for the computer system 100 and is coupled to the processor (102) either bidirectionally (read / write) or unidirectionally (read-only). For example, the storage device (112) may also include computer readable media, such as flash memory, portable mass storage devices, holographic storage devices, magnetic devices, magneto-optical devices, optical devices, and other storage devices. The fixed mass storage device (120) may also provide additional data storage capacity, for example. An example of a mass storage device (120) is an eMMC or microSD device. In one embodiment, the mass storage device (120) is a solid state drive connected via the bus (114). The mass storage devices (112), (120) generally store additional programming instructions, data, etc. that are not normally in active use by the processor (102). It will be appreciated that the information retained within the mass storage devices (112), (120) can be incorporated in a standard manner if desired, as part of the main storage device (110), such as RAM, as virtual memory.

[0027] In addition to providing processor (102) access to the storage subsystem, the bus (114) may also be used to provide access to other subsystems and devices. As shown, these may include a display monitor (118), a communication interface (116), a touch (or physical) keyboard (104), and one or more auxiliary input / output devices (106), including an audio interface, a sound card, a microphone, an audio port, an audio recording device, an audio card, a speaker, a touch (or pointing) device, and / or other subsystems, as needed. In addition to a touch screen and / or capacitive touch interface, the auxiliary device (106) may be a mouse, stylus, tracking ball, or tablet device, and may be used to interact with a graphical user interface.

[0028] The communication interface (116) allows the processor (102) to be coupled to another computer, computer network, or telecommunication network by using a network connection as shown. For example, through the communication interface (116), the processor (102) can receive information, such as data objects or program instructions, from another network, or output information to another network during the execution of method / process steps. Information, which is usually represented as a sequence of instructions to be executed on the processor, can be received from another network and output to another network. Interface cards or similar devices, and appropriate software implemented by, for example, execution / implementation on the processor (102) can be used to connect the computer system 100 to an external network and transfer data according to standard protocols. For example, various process embodiments disclosed herein can be executed on the processor (102), or can be executed in conjunction with a remote processor that shares a portion of the processing, across a network such as the Internet, an intranet network, or a local area network. Throughout this specification, "network" refers to any interconnection between computer components, including the Internet, Bluetooth, WiFi, 3G, 4G, 4G LTE, GSM, Ethernet, TCP / IP, intranet, local area network ("LAN"), home area network ("HAN"), serial connection, parallel connection, wide area network ("WAN"), fiber channel, PCI / PCI-X, AGP, VLbus, Express PCI, Expresscard, Infiniband, ACCESS.bus, wireless LAN, HomePNA, fiber optic, G.hn, infrared network, satellite network, microwave network, cellular network, virtual private network ("VPN"), universal serial bus ("USB"), FireWire, Serial ATA, 1-wire, UNI / O, or any form of connecting homogeneous, heterogeneous systems and / or groups of systems together. Additional mass storage devices not shown may also be connected to the processor (102) via the communication interface (116).

[0029] Auxiliary I / O device interfaces, not shown, may be used in conjunction with computer system 100. Auxiliary I / O device interfaces may include general and custom interfaces that allow processor (102) to send data, and more typically, receive data from other devices such as microphones, touch-sensitive displays, transducer card readers, tape readers, voice or handwriting recognizers, biometric readers, cameras, portable mass storage devices, and other computers.

[0030] In addition, various embodiments disclosed herein further relate to a computer storage product with a computer-readable medium, the computer-readable medium including program codes for performing various computer-implemented operations. A computer-readable medium is any data storage device that can store data that can be read by a computer system thereafter. Examples of computer-readable media include, but are not limited to, all of the above-mentioned media: flash media, such as NAND flash, eMMC, SD, compact flash devices; magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD-ROM disks; magneto-optical media, such as optical disks; and specially configured hardware devices, such as application-specific integrated circuits ("ASICs"), programmable logic devices ("PLDs"), and ROM and RAM devices. Examples of program codes include both: machine code such as generated by a compiler, or files containing higher-level code, such as scripts that can be executed using an interpreter.

[0031] Figure 1 The computer / server system shown in is merely an example of a computer system suitable for use with the various embodiments disclosed herein. Other computer systems suitable for such use may include additional or fewer subsystems. In addition, the bus (114) illustrates any interconnection scheme for linking the subsystems. Other computer architectures with different subsystem configurations may also be utilized.

[0032] Index your data tables. Use indexing to improve the performance of database queries. A typical database index is a data structure that is constructed to quickly locate data without having to search for every row in the database every time the database table is accessed.

[0033] As discussed herein, a subcondition refers to the interpretation of an index value as it is more general than indexing only a specific value in a column. For example, a true value in the index "Sensor 21 High Temperature?" may simply indicate that the corresponding entry in the corresponding row of the table exceeds a specified threshold "Sensor 21 Temperature is greater than 212F." A subcondition such as temperature exceeding a given threshold may be sufficient for the application and may change at a much lower frequency than the actual value. If a query condition specifies a subcondition that is handled by a traditional index, the query processor can access the index and iterate over only those records that the index indicates are relevant to match the rest of the query condition, rather than performing a full table scan on the entire data table associated with the index.

[0034] A full table scan of the entire data table incurs a cost of O(N), where N is the number of rows in the data table. If an index contains K entries, where K is much smaller than N, then a scan of these K entries is much faster than a scan of all N entries, because there are fewer records to check and because the query conditions are reduced by handling the subconditions by the index. In addition, if the average length of each record is B bytes, then a full scan would typically incur N*B bytes of I / O or memory access if the data were in memory, and the index would significantly reduce this I / O or memory requirement.

[0035] Indexes can also be useful for police constraints. For example, a given table must have unique values ​​in column C. If this table is indexed by column C, then when modifying a record or adding a new record, the index can be used to efficiently check whether the value in column C already appears in a record in the table.

[0036] A common traditional technique in database management is to use a bitmap index when the cardinality of the field / column to be indexed is low. For example, in a bitmap index where there are two possible sub-condition values ​​- true and false, the logical bit of each row indicates whether the corresponding entry matches the sub-condition associated with this bitmap. Database sub-conditions often appear as part of a query, such as in a where clause in SQL. If a sub-condition in a query corresponds to a sub-condition represented by a bitmap index, a bitmap index can be used to make the query more efficient.

[0037] For example, if the field is a Boolean value indicating whether an employee is a salaried employee, then the index for non-salaried employees has a true entry for each employee who is not a salaried employee. Then, when a query condition specifies non-salaried employees, the query can determine the rows to consider simply by finding true entries in this bitmap index, thereby evaluating the rest of the query condition for each of these rows. As another example, a sub-condition can be a threshold indicating whether an associated data value exceeds a threshold, e.g., whether an employee was paid more than $50,000 in salary in the last calendar year.

[0038] If the column value / sub-condition has a cardinality greater than a Boolean value, multiple bitmaps can be used together to represent the values. For example, one index can indicate whether the temperature is warm or cool, and another index can indicate whether the temperature is very hot or very cold. Techniques such as binning and thresholding are traditional techniques for using bitmap indexes.

[0039] Traditional indexing. Figure 2 is a diagram of an embodiment of a conventional bitmap index that logically references certain rows in a base data table. Figure 2 As shown in FIG. 1 , an index (202) is associated with a data table (212). An index row (204a, 204b, ..., 204m) corresponds to a particular sub-condition in a data table row (214a, 214b, ..., 214p). Conventionally, the i-th entry of the bitmap corresponds to the i-th row, such as Figure 2 As shown in , there is no need to store the key explicitly.

[0040] One optimization is to use a sparse representation of the bitmap index (202) where only true entries are stored, so any entry not stored in the index (202) defaults to false. Then, if there are relatively few true entries, the bitmap (202) requires significantly less memory and can be traversed faster.

[0041] In this sparse case, the use of a traditional index (202) may be problematic. If the index on column C (216) does not include rows with "don't care" values ​​in column C, then a query matching the "don't care" value in column C cannot use index (202) because index (202) will not include rows that could match. Therefore, the query will require a full scan of the data table (212). On the other hand, if the index (202) on column C (216) includes rows containing "don't care" values ​​in column C, then index (202) will include most of the rows in the table (212), thereby incurring a large space overhead without significantly reducing the query cost compared to a full scan of the data table (212), and given the large number of rows it includes, index (202) again does not provide a significant benefit.

[0042] In the common case of sparse table queries, the query condition may contain multiple sub-conditions, each of which allows matching NULL values. In this case, using the index (202) may be more expensive than simply performing a full scan of the data table (212) because the query using the index (202) will typically scan most of the rows in the table, and most likely in an order different from the order provided by the clustered index, which results in a suboptimal traversal of the indexed rows. That is, the scan may not be performed in the order specified by the clustered index.

[0043] As a related scenario, in some applications, it is expected that queries rely on one or more sub-conditions that are rarely true. For example, if the data of interest is the status of a large number of furnaces in a factory, and the primary query for the furnaces is to check for furnaces that exceed a certain threshold temperature, then having an index (202) that uses the threshold sub-condition is more efficient than indexing the actual temperature. In the case of using the threshold sub-condition, there are typically zero or only a few furnaces that exceed the threshold temperature, where the system does not "care" about the other furnaces. Therefore, the index itself is actually very sparse. In this way, the index may end up being sparse because the data table (212) itself is sparse, or because the index sub-condition is mostly false.

[0044] Alternatively, note in this example that the data table for the oven temperature is sparse and therefore useless to store because the system does not "care" what the temperature is if it does not exceed the threshold. Not storing a value when the system does not "care" about it not only saves space, but also saves repeated updates to the table as the value changes.

[0045] For example, if the temperature readings are updated every 1 second and there are 1000 ovens, there may be 1000 updates to the table per second. On the other hand, if only temperatures exceeding a threshold are stored, there will often be 0 or a small number of updates per second, because either no oven or only a small number of ovens exceed the threshold temperature, and ovens usually do not oscillate between exceeding and falling below the threshold.

[0046] Another area where sparsity comes up is with respect to data sets represented as graphs. Traditionally, graphs represented as tables can be sparse. Traditionally, each node of the graph occupies only a small subset of the set of total edges. A common query includes the sub-condition "nodes directly connected to a given node Ni". In a typical directed graph, a node only shows an indication of its outbound edges, not its inbound edges. Because most nodes are connected to a small subset of the total set of nodes, the index supporting this query is sparse. Furthermore, every node in the graph needs to have such an index, which incurs a significant space cost. For any index, determining the set of nodes that satisfy this sub-condition requires a full table scan of the node table.

[0047] There is a need for means to efficiently query large sparse databases / datasets without incurring significant time and space overhead.

[0048] Indexing mechanism. Figure 3 is a diagram of an index table. Figure 3 As shown in , an index table (302) stores M+1 indexes, one for each column (306a, 306b, ..., 306y) across N rows (304a, 304b, ..., 304z) of the index table (302), and the index table (302) corresponds to a data table (312). The index table (302) is optimized for efficient full scans, while performing multiple index matches as part of the scan. In particular, each column (306) in the index table (302) represents an index. Row (304a) contains an entry value for each of the indexes for a corresponding row in the data table (314a).

[0049] As referred to herein, an index table row (e.g., row #2 (304c)) contains a sub-condition value for each index sub-condition for the data table row corresponding to the index table row (in this example, row #2 (314c)). Note that the sub-condition value may be NULL.

[0050] For example, the sub-condition value Sc2 (309) in row #1 (304b) is the value of the index corresponding to Sc2 of row #1 (314b) of the corresponding data table (312). Similarly, this value is allowed to be NULL, indicating "don't care" or "unknown".

[0051] Figure 4A is a diagram of an index row mask. In one embodiment, the matching mechanism will Figure 3 Takes as input an index table (302) and an index row mask (402) that specifies the value of each index / subcondition (404a, 404b, 404c, ..., 404z), and returns a list of rows that match this mask. The mask can specify a NULL value for an index to indicate "unknown".

[0052] Binary, ternary and quaternary representations. In one embodiment, the value has three or more states, including the values ​​of "true", "false" and "don't care". In one embodiment, a ternary sub-condition value is used so that the sub-condition is represented as a "known" bit and a second "value" bit, the "known" bit indicates known or unknown by a bit value of true or false, the second "value" bit indicates true or false, and the second "value" bit is only interpreted when the "known" bit is set to true. The quaternary terminology using two bits to represent the ternary value is referred to as [a, b] in this article, where a is whether the state is known (0 = unknown, 1 = known), and b is the value associated with the state (0 = false, 1 = true). Under this convention, the interpretation of the allowable [0,1] is: the associated sub-condition "not known to be true": compare [0,0] that can correspond to an unknown state with [0,1] that can be interpreted as "not known to be true". Note that the [0,1] sub-condition in the entry in the index table (302) may match false or unknown inputs, and unlike [0,0], [0,0] does not match any input as true. Thus, [0,1] may not necessarily be considered the same as [0,0], and / or may not be allowed in internal and / or consistent representations. Alternatively, this 4th value may be considered "unknown" and only matched if the corresponding index table entry also indicates the same "unknown" value (i.e., [0,1]).

[0053] In one embodiment, the value has two states, which allows different matching behaviors for different columns of the table. For values ​​with binary states, in many applications, objects have symptoms that the client otherwise "doesn't care about", such as an object that represents isOverheating. Recognizing this, an alternative approach is to have two values ​​per entry instead of three, corresponding to "true" and "don't care". The corresponding match operator output table is:

[0054]

[0055] This reduces the number of bits required per entry to one instead of two - the minimum for a ternary representation, so the memory overhead is only half.

[0056] There are cases where an entity with symptom S1 is needed and a match in a separate rule / root cause that does not have a symptom (such as S1 is false). For this case, an additional symptom S2 corresponding to the negation of S2 can be explicitly introduced, for example, symptom S1 can be lossOfPower (loss of power) and S2 can be hasPower (has power). Then, rows that need symptom S1 can set the corresponding table entry to "true" and set the entry for S2 to "don't care". Conversely, rows that need S2 can set that entry to "true" and set the S1 entry to "don't care". So, for example, if 10% of the symptoms need to be negated, it will still be cheaper in space than the 2-bit approach, that is, an extra 10% of space instead of 100% more space.

[0057] In one embodiment, a combination where both S1 and S2 are true may correspond to "unknown". That is, if the input sets both S1 and S2, the symptom is considered "unknown". In applications, representing the state as "unknown" may be more useful than the interpretation of "not known to be true" for the fourth value. This "unknown" support works without changing or expanding the matching above. Both S1 and S2 entries in a row specify true, and the input also specifies these entries as true, so "unknown" is matched.

[0058] There are also symptoms that are actually discrete enumerations of values, such as very cold, cold, cool, normal, warm, hot, very hot. These seven values ​​can be represented in three bits in a normal binary representation. However, using the "true" / "don't care" binary approach described above, the six symptoms for positive and negative would be needed as separate symptoms for each bit. Alternatively, seven separate symptoms could be used for a "one-hot" representation of these seven values.

[0059] In one embodiment, a specific set of columns used by the symptoms representing this enumeration is used. In this case, specific columns are specified to match "true" / "false" instead of "true" / "don't care" so that the enumeration is represented with three columns instead of six or seven columns. The "don't care" value is still needed so that the row that doesn't care the value of this enumeration symptom can indicate this. Therefore, it is allowed to specify the column value width and reserve a value to indicate "don't care", such as all zeros. In this case, a logical column width of three bits can be specified, so [0,0,0] can correspond to "don't care". Therefore, the enumeration can be represented with three bits, while still allowing "don't care" to be specified in the row that doesn't care the enumeration symptom.

[0060] Therefore, this additional extension allows the width of the logical column to be specified in the table, and the all-zero case is considered "don't care". Thus, the column corresponding to isOverheating or S1, S2 can be marked as one bit wide, while the column corresponding to the above enumeration will be marked as a logical column with three bits.

[0061] In one embodiment, an efficient representation of the column width is to set one bit per column if the next column is a continuation of the column. Likewise, the bit indicating that it is a continuation of the previous column. Thus, the columns corresponding to the enumeration may be labeled [1,1,0] to indicate three-bit columns forming one logical column / entry.

[0062] Typically, a table can specify column widths and matching behavior for a logical column. As an example of this more general case, a table can specify a logical column as five bits wide and matching behavior as a test for "greater than". That is, if the value of the input in these five bits is a binary value greater than the value in the entry of the table, the input can match on the entry, unless the entry of the table is zero and is considered "don't care" and if it is all ones, it is considered "unknown". As another example of specifying matching behavior, matching behavior can be specified as a logical or (OR) of the inputs across these columns. By using a bit in the logical column to indicate whether the symptom is related to the row and excluding the bit from the "or", the all-zero values ​​in the entry of the table can be considered "don't care".

[0063] In one embodiment, a hardware instruction such as the "ANDN" or AND NOT instruction in the Intel and AMD instruction sets can be used with the binary representation. Hardware instructions are typically faster to execute and more efficient to program than their software counterparts. The ANDN instruction performs a bitwise logical AND with an inverted first operand and a second operand, for example, the result mismatch for the two operands inBlock and tableBlock:

[0064] Mismatch=~inBlock&tableBlock

[0065] The corresponding output table is:

[0066]

[0067] Therefore, if the mismatch is complementary to the match, then

[0068]

[0069] and Figure 2 Unlike the traditional index shown in , the disclosed set of indexes is transferred as follows Figure 3, rather than each being considered as a separate / union array of values.

[0070] like Figure 3 As shown in , in a typical case, if there are N rows in the associated data table (312), then the index table (302) has N rows (304). Otherwise, if there is a row (314a...314p) in the data table that does not have a corresponding row in the index table (302), the row can only be found in the query by performing a complete scan of the data table (312).

[0071] Figure 4B is an embodiment of the process of matching operation, which corresponds to the following pseudo code:

[0072]

[0073] Therefore, the disclosed improvement is to use Figure 2 Multiple indexes (202) to scan Figure 3 The N-entry index table (302) is used to avoid Figure 2 and Figure 3 The two data tables shown in (212 / 312) were used to perform a complete scan. Figure 3 The index table (302) of the data table (312) is optimized to be scanned more efficiently than the original base data table (312). That is, the focus is on optimizing the index table scan rather than avoiding N-row data table scans by using conventional indexing schemes. The key focus is on applications where the sparsity of the data table (312) makes it more efficient to use Figure 3 The index table (302) is better than using Figure 2 The conventional index (202) shown in is more efficient.

[0074] Root cause analysis as an application of an index table. One such application where the sparsity of the data table (312) makes it efficient is automatic root cause analysis using ternary failure scenarios. A symptom is referred to herein as a named and / or defined state of a component of a monitored system that is important for distinguishing one failure scenario from another. In one embodiment, a symptom value and a "don't care" value are used, the symptom value corresponding to an "unknown" value corresponding to a symptom value that is not known, the "don't care" value also being referred to as an insignificant value corresponding to a symptom that is not needed for a particular analysis.

[0075] In one embodiment, each symptom value is limited to one of the following: true, false, or unknown. Therefore, the symptom values ​​are referred to herein as "ternary" values. In one embodiment, the unknown value and the "don't care" value are indicated by the same value, and the two are distinguished based on the usage context.

[0076] Overview of Automatic Root Cause Analysis (ARCA). Complex monitored systems can have numerous sources of failure, and even the mechanisms used to monitor such systems are subject to failure. For example, a temperature sensor monitoring a refrigeration system may fail permanently or intermittently, indicating an inappropriate temperature of the monitored system.

[0077] Component dependencies can introduce additional complexity, for example, a cooling coil in a refrigeration system depends on the correct operation of a compressor to provide concentrated refrigerant. These dependencies arise from the interconnection of these components. As described above, a failure in one component may cause another to indicate a fault condition / symptom. Therefore, when one component fails, it may cause cascading failures in components that are dependent on the failed component, making the task of determining the actual root cause failure difficult. In some cases, there may not even be a root cause in the alarm provided to the operator.

[0078] For example, if a cable fails between two computer network switches, there may be a large number of alarms from the switches at either end of the cable. However, there are usually no alarms that directly indicate that the cable is disconnected because there are no sensors directly on the cable that can detect cable breakage. Complex systems can also be implemented in multiple layers, creating another set of dependencies. These layer dependencies are another source of alarms. For example, the above cable failure may cause the transport layer to indicate that it has a session timing failure because no response was received. Similarly, a misconfiguration at the IP layer may cause alarms at the TCP / transport layer and the routing layer to be generated.

[0079] Traditionally, these additional alarms are referred to as symptoms of the root cause failure. Generating a large number of these symptoms as alarms makes it more difficult to determine the actual root cause.

[0080] An efficient way is described to encode the operating principles, correlations and causal relationships, and potential root causes that are known for an engineering system as a result of its engineering design by using efficient symptom matching without the need to use statistical correlations between faults or impractical / costly large training data sets. This efficiency reduces storage costs and / or reduces processor power consumption to determine root cause analysis. This efficient approach allows root cause analysis to be performed automatically and efficiently.

[0081] Symptoms and failure scenarios. Figure 5Ais a graphical representation of a fault scenario vector for a symptom. An example of a symptom, no power, is a symptom that indicates that no power is coming to the monitored system. The state of a symptom can be a known value or a special indication that it is unknown and / or a "don't care". The term "don't care" is commonly used in digital logic to indicate that an associated item is not relevant / not needed. The ability to make a process a "don't care" for a given symptom indication allows analysis to be performed even when that aspect of the system state is not actually known.

[0082] A failure scenario is referred to herein as a set of symptom values ​​that indicate known and unknown failure states of the monitored system. Logically, a failure scenario represents the state and / or potential partial state of a system from the standpoint of an observed / determined sign that something is wrong or nothing is wrong with the system. It may not indicate the full state of the system. For example, in the case of a vehicle, a failure scenario may not necessarily indicate the vehicle's location, speed, etc., only the state of the symptoms, i.e., the aspects required to perform a root cause analysis of the failure.

[0083] like Figure 5A As shown in , in one embodiment, a fault scenario is represented as an array (502) of values, where each entry (504a-m) corresponds to a specified symptom. For example, symptom Sy0 (504a) is the first entry, symptom Sy1 (504b) is the second entry, and so on. In one embodiment, there may be multiple symptoms associated with the same metric. For example, there may be different symptoms for a temperature sensor, namely slightly high, medium high, and extremely high. In one embodiment, there may be symptoms based on different levels of derivatives associated with the same metric. For example, a symptom may be associated with a first-order derivative that is zero for too long, i.e., constant, typically indicating that an input sensor has failed. A symptom may be associated with a first-order derivative that is too high, meaning that it changes too quickly.

[0084] There may be additional symptoms associated with a metric that indicate that the metric is out of range or is behaving inappropriately. In this case, for example, an out of range symptom is set at the same time as a symptom indicating that the metric is too high or too low. This "aggregate" form of symptoms may allow failure scenarios to be specified in terms of "out of range" rather than having to cover both "too low" and "too high".

[0085] Define a match operator between two failure scenarios s0 and s1 to return true if:

[0086] bool isMatching=match(s0,s1);

[0087] If each symptom entry in s0 is either "don't care" or otherwise matches the value in the corresponding entry in s1. Note that the matching operation is not commutative, match(a,b) may not necessarily be equivalent to match(b,a).

[0088] Root cause table. Figure 5B is a diagram of a root cause table (RCT). An RCT is a table where each row is a fault scenario annotated with an associated root cause. In this context, unknown values ​​for symptoms in such a fault scenario are interpreted as "don't care". For example, for the root cause "broken engine", the symptoms in this row may be: "noPower" is false, "motorNotRunning" is true, and all other symptoms are indicated as "don't care".

[0089] In one embodiment, the RCT contains one row for each fault or event that can be a root cause, where each row indicates the symptoms that must be true for this to be a root cause, those that must be false, and the remaining set as indicating "don't care." It is important to note that specifying more symptoms as a specific value other than "don't care" beyond the absolute minimum for a given root cause may result in the root cause not being identified or matched because the additional symptoms may not be known or contrary to the one specified for that row. Therefore, it is important to specify the minimum set of known symptoms required to diagnose the system as a specific root cause associated with that row in the table. If a given root cause can have multiple sets of symptom identifications, there are multiple rows in the RCT because there is one row for each set. A given root cause can have multiple corresponding rows because one row corresponds to the minimum set of symptoms and other rows correspond to the minimum set with additional symptoms that provide greater confidence in the root cause. For example, in the event of a power supply failure to a switch, the minimum set may contain only the "lossOfPower" symptom from the switch's current sensor, while an additional row may contain that symptom plus the "lossOfSignal" symptom from the directly attached switch to the failed switch.

[0090] In one embodiment, each RCT row is represented in the same manner as a failure scenario. Thus, it may be referred to herein as a potential failure scenario. Figure 5BAs shown in , the RCT (522) includes k+1 rows (524a-524l), each row associated with a particular root cause having N symptoms per row. For example, root cause #0 is associated with the first row (524a). The values ​​of the symptoms (504a-m) in each row (524a) are different from the other rows (524b-524l), each corresponding to a potential failure scenario for the associated root cause, as indicated by the root causes labeled #0 through #k.

[0091] Failure scenarios determined from the monitored system are referred to herein as actual failure scenarios, as opposed to potential failure scenarios. For a monitored system, there may be multiple actual failure scenarios. An actual failure scenario may be a more detailed failure scenario for a particular subsystem than another. Another source of multiple actual failure scenarios is uncertainty about the failure. For example, one scenario may have a symptom corresponding to a system temperature being too low, while another may have a symptom indicating that a temperature sensor has failed. In the latter case, it may indicate the temperature sensor-related symptom as unknown.

[0092] As described above, ternary symptom values ​​are used such that the symptom is represented as a "known" bit that indicates known or unknown by being true or false, respectively, and a second "value" bit that indicates true or false, which is itself interpreted only if the known bit is set to true. Using a quaternary terminology of [a,b], again the interpretation of the allowable [0,1] is that the associated symptom is not known to be true. Thus, [0,0], which may correspond to unknown, is different from [0,1], which may be interpreted as not known to be true. Note that the [0,1] symptom in an entry in RCT (522) may match the input being false or unknown, unlike [0,0] which simply does not match true. Again, [0,1] may not necessarily be treated the same as [0,0], and / or may not be allowed.

[0093] Figure 5C is a diagram of a 64-bit block representation of known and value bits. In one embodiment, a fault scenario is represented as a block of bits that is partitioned into a sequence of "known" bits and a sequence of value bits. For example, Figure 5C As shown in , one implementation uses 64-bit blocks, where the first 32 bits are "known" bits, and the next 32 bits are value bits. Figure 5C , if the i-th known bit is 1, then the i-th value bit indicates whether the corresponding symptom is true or false; otherwise the actual value is not known, and the i-th value bit is not meaningful. This embodiment allows efficient determination of the "known" bits in a block. It also means that if all symptoms in a block are unknown or "don't care", then there is no need to store the block. That is, the absence of explicit storage of a block is interpreted as the block containing only "don't care" values.

[0094] Root cause analysis. Figure 5D is an illustration of a root cause analysis technique. The actual root cause associated with a given actual failure scenario (552) is determined by matching each row in the RCT (522) using a matching engine (554) and indicating those that match as probable root causes. That is, if a failure scenario matches a row such that each entry is matched by the match(a, b) operator above, then the root cause associated with that row is output as a probable root cause (556) associated with the symptom, as shown in FIG. Figure 5D as shown in .

[0095] This match is essentially a "ternary match", but unlike the ternary matching provided by a ternary content addressable memory (T-CAM), the input fault scenario is also ternary. However, the T-CAM can be used as part of an efficient / hardware system for matching. There may be multiple simultaneous root cause failures in the monitored system. Therefore, it is possible that the match performed matches multiple rows in the RCT, one for each root cause. For example, the engine may fail at the same time as the temperature sensor has failed by indicating a completely unrealistic reading. There may be multiple rows mapped to the same root cause. This handles the following situation: in the case, the root cause failure can be indicated by a different set of symptoms.

[0096] In one embodiment, a row representation does not explicitly store a "don't care" entry. That is, there is no explicit designation of the i-th symptom or representation that is interpreted as a "don't care" for the i-th symptom. In one embodiment, symptoms are aggregated into blocks associated with logical units or components of the monitored system. For example, an embodiment may use the 64-bit blocks of known / valued bits described earlier. Thus, if a component is not associated with a particular root cause, the entire block does not need to be stored. Each row may then require a relatively small amount of storage. Typically, most rows are relatively sparse because only a small subset of symptoms are associated with a particular fault, and therefore, only a small percentage of the row is actually stored, with the rest being "don't care" by default.

[0097] The representation of any fault criteria is achieved by using multiple symptoms. For example, one root cause is proved by very high temperature, another root cause is proved by high temperature, and another is proved by slightly high temperature. That is, for each of these levels, there can be a symptom entry in each row.

[0098] The key element indicates what is known to be false as a symptom, i.e. no fault, and what is known to be true, i.e. a fault exists, while still allowing for the unknown or "don't care". The false case actually filters out symptoms that are due to another cause, such as the compressor not operating, but there is actually no power, which is the root cause. Thus, a subsystem SSi that depends on multiple other subsystems may require all of these other systems to be known to be operating before a fault in the subsystem SSi can be reliably identified as the root cause.

[0099] In one embodiment, the system may record whether any symptoms have changed in the actual fault scenario since it was last matched, and only rematch the actual fault scenario to the RCT if it has changed (522). This check avoids the overhead of rematching when there is no change to the actual fault scenario.

[0100] In one embodiment, the frequency of rematching is configurable according to the application needs. For example, ARCA matching in a refrigeration system can be configured to be performed every 30 minutes to minimize the cost of matching, because failures do not cause immediate problems and the resulting delay in detecting the failure does not significantly affect the interval between repairs. This low matching rate assumes that the following failures are not critical to detect: the failure appears briefly enough and then disappears before matching occurs, and thus does not exist in the actual failure scenario when matching.

[0101] RCT level. Figure 5E is an illustration of an RCT hierarchy. In one embodiment, there may be different levels of RCT hierarchy corresponding to root cause analysis. For example, the refrigeration system RCT (522) may be partitioned so that it has a top-level RCT (572) that "roots causes" the high-level reasons why the system is not performing well, which may be one of the following: 1) not maintaining the desired temperature, and 2) consuming excessive energy. Once the top-level RCT (572) indicates one of these reasons, the next level of monitoring and corresponding RCT (574) may be automatically deployed depending on the specific cause, thereby providing a more specific root cause for the top-level root cause. For example, if the top-level root cause for poor performance is excessive energy consumption, additional telemetry and analysis may be deployed to detect symptoms such as low coolant, ice on the cooling coil, overcycling of the compressor, failure of the current sensor, and other possible root causes of excessive energy consumption. This next level root cause may be sufficient to indicate the necessary repair of the system. Alternatively, based on the root cause determined at this second level, the next level of RCT (574) and processing may be automatically dispatched.

[0102] This hierarchical processing can reduce the resources consumed by the root cause analysis in situations where the system is operating normally. It can also reduce the resources required to determine the root cause of a particular failure if the next level of root cause analysis only needs to deal with a subset of possible symptoms based on the indication of the higher level root cause. For example, using the above case of a refrigeration system, it is known that the problem in this system case is excessive power consumption. The actual deployed next level root cause analysis processing may require a smaller RCT (574) and less telemetry and processing than this level of root cause analysis configured to detect both failures to maintain the configured temperature and excessive power consumption. Alternatively, if both top-level symptoms are present, there may be no such savings. However, it is feasible to run two instances of this detailed root cause analysis in parallel as separate processes, which is time efficient.

[0103] Typically, root cause analysis matching can be performed in parallel by partitioning the RCT across multiple parallel threads and collecting the output of each. Since matching does not modify the actual failure scenario or RCT, and since matching is order-independent across rows, the only synchronization required between threads is on the output of the aggregated root cause set.

[0104] Multiple layers, partitions, and this hierarchical approach significantly reduce the size of the RCT. For example, in a network application, if a higher level RCT, such as basic connectivity, only considers four symptoms per network node, instead of 256, the RCT can be reduced to about 1 / 64 in size. This size can be further reduced by having only coarse-grained root causes in the basic RCT. For example, a large number of specific problems for links can be handled as root causes at this level by a simple "link problem", which when identified can cause the assignment of a more detailed ARCA, using the full set of possible specific link problems.

[0105] In addition to reducing the size of the RCT that needs to be actively accessed in the common case where there are no faults, the smaller RCT is more efficient to process. Moreover, with enough reduction in size, the fault scenario vectors can potentially fit in a hardware T-CAM so that the matching can be done in hardware. In one embodiment, where there are multiple similar independent units for root cause determination, such as in the case of an HVAC application with multiple rooftop units (RTUs), a single RTU RCT can be used to sequentially pass the fault scenario vectors for each separate RTU into the T-CAM to detect faults in each RTU.

[0106] A benefit of such a hierarchical approach is that basic ARCA is expected to be the common case, i.e. when the equipment is performing properly, so T-CAM can be very efficient in handling the common case, and detailed ARCA can be reserved for when there is actually a problem. Another benefit is that the more basic ARCA allowed by the hierarchical approach can mean less telemetry is collected when there are no faults, which again may be the common case. Thus, a hardware implementation, for example using T-CAM or equivalent, which can currently be implemented using SRAM, is practical and attractive in some applications.

[0107] ARCA and index tables. Compare Figure 3 and Figure 5B As can be seen, the index table (302) corresponds to a set (522) of potential failure scenarios, with each row (524) containing one potential failure scenario. The symptoms (504) correspond to sub-condition entries in the table. For example, where the data table (312) consists of one row (314) for each furnace in the plant and the entries correspond to furnace temperatures, the index (306) corresponding to "temperature exceeds threshold" will set to true the entry in the column of the index table (302) corresponding to this sub-condition for each row (304) in which the corresponding furnace exceeds the temperature threshold. Thus, for a given symptom (e.g. Figure 5B The entries in the set of potential failure scenarios for Sy2 in correspond to the indices in the index table (302) and thus the columns (306). Figure 5D The output / possible root cause (556) corresponds to the matched row in the general index table approach. The actual failure scenario (552) corresponds to the mask (402) in the general index table approach. And, the NULL value is a special value corresponding to "don't care" or "unknown", where NULL is a term traditionally used in database literature. With this translation of terminology, the matching method (554) used in ARCA is similar to the matching method used in the index table (302), where the matching logic / comparison between the mask and the row is inverted in query usage.

[0108] In one embodiment, one or more of the indexes in the table are bitmap indexes that are extended with the ability to indicate a "don't care" value as well as true and false. This corresponds to the ternary or quaternary matching described above. The minimum representation of a ternary or quaternary representation and a match is two bits, i.e. using the known / value representation of the original application, such as Figure 5C As shown in the figure.

[0109] Figure 6 is a diagram of the query process using the index table. Figure 6 As shown in , the query process using the index table is similar to Figure 5DARCA technology. A query statement (602) is received, where the query processor (604) processes the statement to form an index mask (402). The matching engine (608) then matches the index mask (402) with the index table (302) to produce matching rows (612) as output. The query processor (604) then provides additional filtering (614) to process the query results (616). For example, the query may request all engineering salaried employees hired within the past 6 months with a salary of more than $50,000. If the table supports sub-conditions corresponding to isSalaried, isEngineer, and isOver500000, the additional filtering will check in each matching row whether the hire date is within the past 6 months, thereby removing each matching row that fails the check from the matching set.

[0110] Therefore, when comparing Figure 6 and Figure 5D ARCA using a ternary failure scenario representation is a specific application of the index table (302). In particular, root cause analysis can be viewed as being implemented as a query (602) to the index table (302) / RCT (522) to determine one or more rows that match (554, 608) the set of symptoms specified in the index row mask (402) / actual failure scenario (552), and then mapping each of these rows to a corresponding matching row (612) / possible root cause (556). The only difference is that in the case of a database query, the comparison is inverted.

[0111] In ARCA applications, there may be no need to reference the actual data table of telemetry as part of responding to queries. In fact, there may not even be a need to maintain a data table of incoming telemetry, but rather to respond to each telemetry update by simply setting or clearing the symptoms in the actual failure scenario (if no changes are required). Furthermore, since the sub-conditions, also called symptoms, are selected based on those conditions required to determine the root cause failure, the index table (302) itself is typically extremely sparse.

[0112] Rule condition evaluation in rule-based system applications. Another common application area of ​​index tables is for evaluating rule conditions in rule-based systems. The challenge in this scenario is to determine which rule conditions are actually true in order to decide which rule actions to execute. An index table (302) is used by having one row (304) per conjunctive rule condition for each rule in the rule set of interest. Each row (304) specifies the sub-condition values ​​for this rule, using NULL and / or "don't care" for those sub-conditions that are not relevant to the rule. Each such row (304) directly or indirectly references the corresponding rule action. In this use case, the matching operations are applied in the same order as ARCA.

[0113] If a rule contains a condition that is a disjunction of two sub-conditions SC0 and SC1, there is a row corresponding to SC0 and a row corresponding to SC1, both of which map to the same rule action. The matching operation performed on the index table (302) using the mask (402) indicating the current sub-condition then provides a reference (612) to the rules whose conditions have been satisfied, allowing these actions (614) to be further refined and then executed. In this way, the index table supports efficient determination of which rule conditions are true, allowing the rule-based system to execute correctly. The query form of matching can be used on the same table to support queries on rule conditions (e.g., all rules that apply when the temperature is "isTooHot").

[0114] refer to Figure 3 , in target application domains with sparse data, a full scan of the index table (302) is less expensive than a full scan of the data table (312) for several reasons:

[0115] The size of the index table (302) is smaller than the size of the data table (312) because Figure 5C As shown in , each entry may have only two bits, rather than the full value of 32 or 64 bits or even larger in the case of a descriptor, array, or string value;

[0116] The index table (302) can be constructed or arranged in memory to more easily optimize scanning because it is smaller and changes much less frequently in many cases of sub-conditions;

[0117] The sub-condition index table (302) can efficiently represent the sub-conditions of interest, such as temperature exceeding a threshold, without requiring the application to calculate the sub-condition. Thus, the number of memory accesses and the amount of computation required to check the sub-conditions is reduced.

[0118] As a simple example, a threshold requirement is checked using raw data values ​​without subconditions indicated:

[0119] 1. Read the current value and the threshold value;

[0120] 2. Check if the current value is known;

[0121] 3. If it is not known, branch;

[0122] 4. Compare the two values; and

[0123] 5. Branch based on the comparison results.

[0124] In contrast, checking a threshold condition in the index table simply tests the associated 2-bit value. For example, if the sub-condition is "isTooHot" corresponding to a current temperature value exceeding the configured threshold, then: if the corresponding input entry is [1,1] indicating that the sub-condition is true, then a match only requires checking that the corresponding entry in the row is also [1,1]. In addition, for Figure 5C , or similar, and a 64-bit known and 64-bit value, the evaluation is performed in the match operation as part of evaluating 64 such subconditions at a time, and is therefore nearly 100 times more efficient.

[0125] refer to Figure 3 In one embodiment, each index table row (304) has an associated set of pointers or IDs that reference rows (314) or objects in (one or more) base (312) data tables or data sets. In this case, when Figure 6 When the matching operation (608) shown in matches on a row, it adds the contents of the per-row collection to the result set (612) it is generating. The collection can also be a regular index of these rows on the secondary sub-condition. In this case, if the secondary sub-condition is known, only those rows in the secondary index collection that satisfy the secondary sub-condition for the query are added to the result set (612). Using this per-index row collection, the index table (302) can also be constructed to handle multiple data tables or data sets, similar to database clustering. In this case, the rows (304) in the index table (302) can explicitly indicate the table as well as the row in its associated collection for those rows or objects associated with the matching index table row, thereby allowing multiple tables.

[0126] A query with a given description can then return rows from either or both tables. This aggregation of indexed rows can reduce the number of indexed rows when multiple rows match the same index subcondition (whether in the same table or in different tables).

[0127] In one embodiment, the matching operation is parallelized by partitioning the index table (302) into K partitions by row range, running parallel sub-matching threads on each partition, and then merging the results from each thread to produce the final matching result. This parallel execution is possible because the matching operation (608) does not change the index table (302) or the mask (402), and there is no dependency on the order in which each row is matched. Therefore, different threads can perform matching on different row ranges in parallel.

[0128] Because of its relatively simple processing, the matching operation (608) can be optimized in numerous other ways, including by performing the matching in dedicated hardware, as described above with respect to T-CAM.

[0129] Figure 7 is a diagram of a preferred layout of an index table. In software, the index table (302) can be formatted in memory to minimize cache references. In particular, if the common case of mismatches requires looking at the first K sub-conditions on average, the table can be formatted as a contiguous memory region divided into blocks sized to accommodate K sub-condition values ​​plus a pointer to an extended block.

[0130] The format is Figure 7 The middle diagram shows that there are three sub-condition entries in the initial part of the row (702), and there are pointers to the extension of the row (704). Using this layout, the scan will access memory sequentially for most of the row because there is a mismatch before the extension must be accessed, which allows the processor prefetch mechanism to work effectively, thereby reducing latency and memory bandwidth.

[0131] In some applications, the index table (302) may be optimized by essentially sorting it into a clustered index based on certain sub-conditions. This constrained row range is specified as input to the matching operation in the pseudo-code above.

[0132] Similarly, if the data table (312) logically contains T different object types, the index table can be organized into T different ranges. Then, when the type of the object to be matched is known, the matching operation can first determine the relevant row range, and then perform the matching operation only on the row range, thereby reducing the number of index table entries that need to be scanned. In some applications, it is feasible to further sort the rows in a range into sub-ranges corresponding to another sub-condition, thereby further constraining the range of rows that need to be matched. It should be noted that the cluster indexing is applied to the index table (302) rather than to the base data table (312) as traditionally done.

[0133] Figure 8is a flow chart illustrating an embodiment of a process for querying data records using an index table. In one embodiment, Figure 8 The process of Figure 6 Systems in and / or Figure 1 to be executed by the computer in the system.

[0134] In optional step (802), a set of queries for a set of data records (e.g., in a data table) is received. In step (804), reference is made to Figure 3 , determine a set of sub-conditions (306a, 306b, 306c, etc.) required to support a set of queries to the data table (312). For example, a sub-condition corresponding to "temperature exceeds threshold" for a system with frequent queries about temperature monitoring will have the following entry set to true: the entry in the column corresponding to the sub-condition in the index table (302) for each row (304) in which the corresponding stove exceeds the temperature threshold. In one embodiment, this includes determining the columns in the index table. For example, the index table can be a collection of multiple indexes (306a), (306b), (306c), etc.

[0135] return Figure 8 , in step (806), an index table (302) is maintained. In particular, the index table is stored and updated as appropriate, wherein each entry in the index table indicates the state of a sub-condition associated with a column. In one embodiment, the state type of the state of the sub-condition is ternary. In one embodiment, the state of the sub-condition includes a "don't care" state. In one embodiment, the state of the sub-condition includes at least one of the following states: "don't care", true, and false. For example, for a sub-condition corresponding to "temperature exceeds a threshold", the entry is set to true when the temperature exceeds the threshold, is set to false when the temperature does not exceed the threshold, and is set to "don't care" if the state is unknown or the state is "don't care". Therefore, the index table (302) can efficiently represent the sub-condition of interest (such as temperature exceeds a threshold) without the need for the application to calculate the sub-condition. Therefore, the number of memory accesses and the amount of computation required to check the sub-condition are reduced. In one embodiment, the same state type is used in a given column. In one embodiment, the state type is a representative numeric value. In one embodiment, this includes maintaining the index table with an index record for each data record.

[0136] In step (808), a specific query is received indicating the status of a specified set of sub-conditions. Figure 6, receiving a query statement (602) for processing by a query processor (604). For example, the query processor (604) can obtain the set of queries and process them into an index mask (402). In step (810), the specified set of sub-condition states (402) are matched with the index table (302). For example, the matching engine (608) matches the sub-conditions with entries in the index table based on the above pseudo code. In step (812), a result set corresponding to the matching rows (612) in the index table (302) that match the specified set of sub-condition states is returned, for example as a query result (616). In one embodiment, Figure 8 An optional step not shown in the diagram includes updating the status field in the index table to reflect the status of the affected sub-conditions when new data is received. In one embodiment, the index table rows are sorted by the sub-condition criteria. Therefore, the matching operation on the sorted table can be limited to the corresponding row range, thereby reducing the cost of matching, for example, if you are looking for someone who is 44 years old, you can browse between the row range of 40 to 50 years old.

[0137] In summary, using an index table to support a set of queries on sparse data exploits the efficiency of triple matching, which has the advantage of reducing processor power and query response time, without having to parse indexes for NULL values, which has the advantage of reducing memory space requirements.

[0138] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, the present invention is not limited to the details provided. There are many alternative ways to implement the present invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system for a sparse data index table, comprising: processor; as well as a memory coupled to the processor, wherein the memory is configured to provide instructions to the processor that, when executed, cause the processor to: determining a set of sub-conditions required to support one or more queries on a set of data records; maintaining an index table, wherein entries in the index table indicate sub-condition states associated with data records in the set of data records; receiving a specific query indicating a specified set of sub-condition states, wherein the specific query involves a row range and an index mask of the index table; Matching the specified group of sub-condition states with the index table includes: For a mask entry in the index mask and a corresponding entry for an index row in the row range: comparing the mask entry to the corresponding entry, wherein the mask entry specifies a value of one or more sub-condition states, and wherein the mask entry relates to true, false, or unknown; and In response to determining that the mask entry does not match the corresponding entry, omitting comparison of other mask entries of the index mask with other entries of the index row; and Returns a result set corresponding to the matching rows in the index table that match the specified set of sub-condition states. If a specific index row entry is in a “don't care” state and a specific corresponding entry of the specified set of sub-condition states is in a “true” state, then the specific index row entry is considered to match the specific corresponding entry.

2. The system according to claim 1, wherein: Maintaining the index table includes maintaining the index table with an index record for each data record.

3. The system according to claim 1, wherein: The processor is further configured to update a status field in the index table to reflect a status of an affected sub-condition when new data is received.

4. The system according to claim 1, wherein: The sub-condition states include a "don't care" state.

5. The system according to claim 4, wherein: The state type of the sub-condition state is binary.

6. The system according to claim 4, wherein: The state type of the sub-conditional state is ternary.

7. The system according to claim 4, wherein: The sub-condition states include at least one of the following states: "don't care", true, and false.

8. The system according to claim 1, wherein: Use the same status type in all columns of the index table.

9. The system according to claim 8, wherein: Status types include representative values.

10. The system according to claim 1, wherein: The index table rows are sorted according to the sub-condition criteria.

11. A method for a sparse data index table, comprising: receiving a set of queries for a collection of data records; determining a set of sub-conditions required to support the set of queries; maintaining an index table, wherein entries in the index table indicate sub-condition states associated with data records in the set of data records; receiving a specific query indicating a specified set of sub-condition states, wherein the specific query involves a row range and an index mask of the index table; Matching the specified group of sub-condition states with the index table includes: For a mask entry in the index mask and a corresponding entry for an index row in the row range: comparing the mask entry to the corresponding entry, wherein the mask entry specifies a value of one or more sub-condition states, and wherein the mask entry relates to true, false, or unknown; and In response to determining that the mask entry does not match the corresponding entry, omitting comparison of other mask entries of the index mask with other entries of the index row; and Returns a result set corresponding to the matching rows in the index table that match the specified set of sub-condition states. If a specific index row entry is in a “don't care” state and a specific corresponding entry of the specified set of sub-condition states is in a “true” state, then the specific index row entry is considered to match the specific corresponding entry.

12. The method according to claim 11, wherein: Maintaining the index table includes maintaining the index table with an index record for each data record.

13. The method of claim 11, further comprising updating a status field in the index table to reflect the status of the affected sub-conditions when new data is received.

14. The method according to claim 11, wherein: The sub-condition states include a "don't care" state.

15. The method according to claim 14, wherein: The state type of the sub-condition state is binary.

16. A computer program product embodied in a non-transitory computer-readable storage medium and comprising computer instructions for: receiving a set of queries for a collection of data records; determining a set of sub-conditions required to support the set of queries; Maintain an index table where, An entry in the index table indicates a sub-condition status associated with a data record in the set of data records; receiving a specific query indicating a specified set of sub-condition states, wherein the specific query involves a row range and an index mask of the index table; Matching the specified group of sub-condition states with the index table includes: For a mask entry in the index mask and a corresponding entry for an index row in the row range: comparing the mask entry to the corresponding entry, wherein the mask entry specifies a value of one or more sub-condition states, and wherein the mask entry relates to true, false, or unknown; and In response to determining that the mask entry does not match the corresponding entry, omitting comparison of other mask entries of the index mask with other entries of the index row; and Returns a result set corresponding to the matching rows in the index table that match the specified set of sub-condition states. If a specific index row entry is in a “don't care” state and a specific corresponding entry of the specified set of sub-condition states is in a “true” state, then the specific index row entry is considered to match the specific corresponding entry.

17. The computer program product of claim 16, wherein: Maintaining the index table includes maintaining the index table with an index record for each data record.

18. The computer program product of claim 16, further comprising updating a status field in the index table to reflect the status of an affected sub-condition when new data is received.

19. The computer program product of claim 16, wherein: The sub-condition states include a "don't care" state.

20. The computer program product of claim 19, wherein: The state type of the sub-condition state is binary.

Images