List-based data search using an append-specific data structure

Abstract

The present invention relates to a computer-implemented method for performing database queries in a database (104), said database being physically stored in the form of field-specific data value lists (116), said method comprising the steps of: receiving an instruction to modify a data value; storing (604) said instruction in an append-only data structure (202) without making said modification to said field-specific data value lists (116), said instruction being stored in an append-only data structure (202) without making said modification to said field-specific data value lists (116), said append-only data structure having each entry, referred to herein as an AOD entry, containing at least a field identifier-data value pair of one of said data records that is to be modified according to one of said modification instructions; storing (606) for each of said data records for which the database receives an instruction, in an address allocation table (226), linked to the data record ID of said data record, an address (206, 208) of the most recent of the stored AOD entries specifying the modification to that data record; performing a database query, said database query being: i. searching (612) the field-specific data value list to identify IDs of data records (214); ii. evaluating (614) the address assignment table to identify addresses of AOD entries assigned to one of the data record IDs identified in i); iii. accessing (616) the identified addresses of the AOD entries; and iv. using the change details contained in these identified AOD entries (618) to add field identifier-data value pairs to the data record IDs identified in step i) and outputting same.

Claims

1. A computer-implemented method for performing a database query within a database (104), wherein the database includes a plurality of logical data records at a first point in time called the “most recent integration point,” each data record includes a data record ID and one or more field identifier-data value pairs, the data records are physically stored in the form of a field-specific data value list (116,402,404,406), and the method after the most recent integration point: - A step (602) of receiving a change command to change the data values ​​of fields in some of the data records; - A step (604) of storing the modification instructions in an append-only data structure (202) without making the modifications to the field-specific data value list (116), wherein each entry in the append-only data structure, referred to here as an AOD entry, includes at least one field identifier-data value pair of one of the data records that should be modified according to one of the modification instructions; - Step (606) the database stores in an address assignment table (226) the address of the most recent stored AOD entry specifying the change to each of the data records for which, after the most recent integration time, one or more instructions have been made to change the data value of that data record, linked to the data record ID of that data record, wherein the link in the address assignment table is automatically updated; - The step of performing a database query, the database query is: i. Search the field-specific data value list (612) and identify the ID of the data record (214) based on a match with one or more field-specific search values; ii. In order to identify the address of the AOD entry assigned to one of the data record IDs identified in i), access the address assignment table (614); iii. Access the identified address of the AOD entry (616); iv. Using the change details contained in these identified AOD entries (618), add the field identifier-data value pair to the data record ID confirmed in step i) and output it, The method of computer implementation.

2. - There is exactly one entry in the address assignment table that corresponds to each of the logical data records, and / or The method further includes coordinated management by the DMS system of the logical data records and the entries in the address assignment table, wherein the logical data records are always generated such that the ID of each logical data record explicitly or implicitly specifies the memory address of its entry in the address assignment table, and are synchronized with the address assignment table, in which the ID of the logical data record is assigned the address of the AOD entry having the most recent change to that data record. The computer-implemented method according to claim 1.

3. - A step of integrating the field-specific data value list at a second point in time called the “new integration point,” independently of and / or in parallel with the execution of a database query against the field-specific data value list, thereby integrating the changes directed since the most recent integration point; or - At a second point in time, referred to as the “new integration point,” the process includes a step of integrating changes directed since the most recent integration point by generating an integrated copy (228) of the field-specific data value list, independently of and / or in parallel with the execution of a database query against the field-specific data value list. The computer-implemented method according to claim 1.

4. - For each data value in each field-specific data value list, the data record IDs of all logical data records to which that data value is no longer assigned for that field as a result of change instructions received between the first and second integration points are stored in a list called the negative list (220); The further step includes, for each data value in each field-specific data value list, and for each new data value in each field, storing in a list called a positive list (218) the data record IDs of all logical data records to which that data value will be assigned for that field as a result of change instructions received between the first integration point and the second integration point, Performing the integration includes updating the data values ​​and data record IDs of the field-specific data value list or a copy of the field-specific data value list in accordance with the contents of the positive list and the negative list. After integration, the positive list and the negative list will be emptied. The computer-implemented method according to claim 3.

5. - The method includes, without interruption, in parallel with and independently of the integration, performing further database queries against the field-specific data value list, and / or - The field-specific data value list does not have any locks on the data values ​​included in the field-specific data value list during the execution of the integration. The computer-implemented method according to claim 3.

6. Performing the integration for each of the aforementioned field-specific data value lists means: - The steps include analyzing field-specific and data value-specific positive lists (218) and negative lists (220) of data record IDs to determine the change in the number of unique data values ​​for a given field and the change in the maximum number of data record IDs linked to the data values ​​for that field between the first and second integration points; - A step of determining the number of unique data values ​​included in the integrated version of the field-specific data value list as a function of the change in the number of unique data values ​​confirmed in the above analysis; - A step of determining a first memory requirement for storing the maximum number of data record IDs confirmed in the analysis, which are assigned to the data values ​​in the integrated version of the field-specific data value list; and a second memory requirement for storing the maximum data value stored in the integrated version of the field-specific data value list; - A step of calculating the memory requirements required to store the unified version of the field-specific data value list as a function of the number of unique data values ​​in the list and the first and second memory requirements; - A step of verifying a contiguous region on a physical data carrier that is at least the same size as the calculated memory region; - A step of generating the integrated version of the field-specific data value list by incorporating the changes stored in the positive list and the negative list since the last integration; - The step of storing the integrated version of the field-specific data value list in a verified contiguous area of ​​the data carrier, The computer-implemented method according to claim 3.

7. Further including the provision of the field-specific data value list, wherein the provision is: - A step of parsing raw data to create original data records, wherein each original data record includes one or more pairs of field identifiers and the original data values ​​assigned to them, in addition to a data record ID; - A step of storing a non-redundant, field-specific list of original data values ​​in the database, wherein each original data value in one of the non-redundant, field-specific list of original data values ​​is assigned all the data record IDs of the original data records that contain the original data value in the field representing the list of original data values; - A step of generating a mapping table (210) which assigns to each original data value in the non-redundant original data value list at least one mapping ID that is not assigned to any other original data value; - A step of converting the original data record into the plurality of logical data records, and converting the non-redundant field-specific original data value list into the field-specific data value list, wherein the conversion includes replacing the original data values ​​with mapping IDs according to the mapping table, and the data values ​​assigned to the fields of the data record are the mapping IDs. The computer-implemented method according to claim 1.

8. The computer-implemented method according to claim 7, wherein the mapping ID is a value whose length and / or type is selected depending on the processor architecture of the computer system used for database retrieval.

9. At least one of the modification instructions is an instruction to change or delete an outdated data value of a field in at least one of the logical data records, and the method is: The process further includes the step of storing the data record ID of at least one data record in a list of data record IDs called a negative list (220), the negative list being stored in a data structure (216) linked to a field identifier of one field and linked to a data value to be changed or deleted in accordance with the change instruction. The execution of the database query for each of the field-specific search values ​​is as follows: - Check whether the data structure (216) contains a field-specific search value and a negative list stored linked to the same data value and field identifier as the field identifier of the search value; If so, calculate the difference between all the data record IDs confirmed in step i) for the search value specific to this field and the data record IDs in the negative list; - Including using the aforementioned difference amount of data record IDs for steps ii to iv, The computer-implemented method according to claim 1.

10. At least one of the modification instructions is an instruction to assign a new data value to a field in at least one of the data records, and the method is: - A step of storing the data record ID of at least one data record in a list of data record IDs called a positive list, wherein the positive list is linked to the field identifier of one field and linked to the new data value and stored in a data structure (216); - When the new data value replaces an outdated data value, the step of storing the data record ID of at least one data record in a list of data record IDs called a negative list (220), wherein the negative list is linked to the field identifier of one field and linked to the outdated data value, and is stored therein. The execution of the database query for each of the field-specific search values ​​is as follows: - Check whether the data structure includes a field-specific search value and a positive list stored linked to the same data value and field identifier as the field identifier of the search value; - If so, calculate the union of all data record IDs confirmed in step i) for the lookup value specific to this field and the data record IDs in the positive list, wherein if the new data value replaces an old data value, the data record IDs in the union are reduced by the number of data record IDs in the negative list assigned to this lookup value and that field; - Including using the aforementioned union of data record IDs for steps ii to iv, The computer-implemented method according to claim 1.

11. The data structure (216) includes a searchable sorted array of elements, the array being a list of list elements or a search tree of nodes. The array represents one of the fields in each case, - The elements of the array represent one data value from a non-redundant list (116) of data values ​​included in the data record (214) and assigned to the field represented by the array, Each element of the array is stored linked to an empty or non-empty positive list and / or an empty or non-empty negative list. The computer-implemented method according to claim 7.

12. Each of the field-specific data value lists (116) is a non-redundant data value list in which data values ​​assigned to that field are selected to represent that field-specific data value list in the logical data record. Each data value in each field-specific data value list is unique and is stored linked to the data record ID of all logical data records in that field that contain that data value, represented by that field-specific data value list. The computer-implemented method according to claim 1.

13. The aforementioned integration is: - At the second point in time mentioned above, the step of receiving a command to integrate changes instructed since the most recent integration point; In response to receiving the aforementioned command: - Implement the changes in the field-specific data value list or a copy thereof that were instructed between the most recent integration time and the second time, thereby generating an integrated field-specific data value list (228), such that each data value in each integrated field-specific data value list is assigned only the ID of the logical data record containing that data value, even after taking into account the changes in that field that were instructed between the first time and the second time; - To perform a database search after the second point in time, use the integrated field-specific data value list instead of the previously used field-specific data value list (216); - Use the second point in time mentioned above as the new, most recent integration point. Steps including The computer-implemented method according to claim 3.

14. In response to receiving the aforementioned command, after generating the integrated field-specific data value list: The step further includes regenerating the at least one data structure (216) based on the integrated field-specific data value list, The computer-implemented method according to claim 13.

15. - At least a portion of the logical data record includes one or more "is-superordinate-to" fields, each of which is configured to store the data record ID of a lower data record of that data record; - The field-specific data value list includes an "is-superordinate-to" data value list (416) representing the "is-superordinate-to" field, the data values ​​stored in the data value list being IDs of subordinate logical data records of at least one other logical data record, and each data value in the "is-superordinate-to" data value list being assigned one or more IDs of the other subordinate data records; The database query includes a completeness search parameter that specifies whether, in addition to the data records confirmed in the database query, subordinate data records of these data records are also output; The execution of the aforementioned database query is as follows: It was decided that the aforementioned lower-level data records should also be output; - To obtain one or more IDs of data records below the data record identified in step i), search the "is-superordinate-to" data value list using the data record ID identified in step i); - Evaluate the address assignment table to identify the address of the AOD entry assigned to one of the IDs among the identified lower data records; - In order to add subordinate data records to the data records confirmed in the database query, this includes accessing these identified addresses of the AOD entries. The computer-implemented method according to claim 1.

16. A field-specific data value list includes multiple field-specific data value lists called time value lists, each time value list consisting of a non-redundant list of time points, each time point representing the point in time when the validity of the data value of the field to which the time value list relates begins or ends in one or more logical data records. Each time point in the aforementioned time value list is assigned the ID of the logical data record that is valid at that time; The method in question is: - A step of generating a new version of a data record to be changed in response to receiving a change request for one of the logical data records, wherein the new version is a new logical data record that includes at least one previous version field, the previous version field includes the ID of the data record to be changed, and the new version, rather than the data record to be changed, includes the change and change time specified in the change request; - A step of storing the new version of the data record having a new data record ID in the field-specific data value list, and storing the start of validity of the new data record in the time value list and the field referenced by the change instruction, wherein the ID of the new version is stored in the time value list, The computer-implemented method according to claim 1.

17. - The AOD entries are stored as elements of the blockchain within the append-only data structure, linked together via cryptographic hash values. - The execution of the database search includes validity checks of the hash values ​​of AOD entries processed during the database query process. The computer-implemented method according to claim 1.

18. The computer-implemented method according to claim 1, wherein the database query is performed by a data processing and retrieval system—DMS system (102)—configured to manage and persist the logical data records in the form of a field-specific data value list, and the DMS system is configured not to manage and persist any data structure in addition to the field-specific data value list, from which all logical data records already managed by the DMS system at the first integration can be reconstructed.

19. The computer-implemented method according to claim 1, wherein the field-specific data value list is free from locks ("database locks") on any or all of the field-specific data value lists retrieved in step i), and is free from locks on the data values ​​contained therein, at least during the execution of step i).

20. A volatile or non-volatile storage medium storing computer-readable instructions, wherein the instructions are configured to cause a processor to execute a method for performing a database query within a database as described in any one of claims 1 to 19.

21. A computer system (100,500): • At least one processor (108); - A data memory including a database (104), wherein the database includes a plurality of logical data records at a first point in time called the “most recent integration point,” each data record includes a data record ID and one or more field identifier-data value pairs, and each data record includes a data record ID and one or more field identifier-data value pairs, and the data records are physically stored in the form of field-specific data value lists (116), the database (104); - A data processing and retrieval system—DMS system (102)—the DMS system is configured to manage the database (104), and the management is performed after the most recent integration point: - The step of receiving a change command to change the data values ​​of fields in some of the data records; - A step of storing the instruction in an append-only data structure (202) without making the aforementioned changes to the field-specific data value list (116), wherein each entry in the append-only data structure, referred to here as an AOD entry, includes at least one field identifier-data value pair of one of the data records that should be changed according to one of the change instructions; - The step of the database storing in an address assignment table (226) the address of the most recent stored AOD entry specifying the change to each of the data records for which, after the most recent integration time, one or more instructions have been made to change the data value of that data record, linked to the data record ID of that data record, wherein the link in the address assignment table is automatically updated; - The step of performing a database query, the database query is: i. Search the field-specific data value list (612) and identify the ID of the data record (214) based on a match with one or more field-specific search values; ii. To identify the address of the AOD entry assigned to one of the data record IDs identified in i), evaluate the address assignment table (614); iii. Access the identified address of the AOD entry (616); iv. Using the change details contained in these identified AOD entries (616,618), add the field identifier-data value pairs to the data record IDs identified in step i) and output them, Computer system.

Images