Query planning E

Abstract

There is provided a data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which ea

Description

This invention relates to a data storage querying system, particularly one that stores data and enables the computationally efficient querying of that stored data to provide results to user queries. DESCRIPTION OF THE PRIOR ART We will first describe a conventional data management system that responds to queries from a large number of users; the system ingests data from thousands of external data sources, and then aggregates and stores that data in a datastore. This general context is ubiquitous and arises in many different computer systems, for example, web search systems, financial data systems, flight search systems, LLM-based AI systems with RAG (retrieval augmented generation) and many others. A highly simplified schematic is shown in Figure 1. Moving through this schematic: Multiple end-users 1 make data requests or user queries at their data terminals (e.g. personal computers, mobile phones etc). For some systems, the number of user queries can be in the thousands per minute, or more. These queries are highly variable and often require accessing data originating from multiple (e.g. over 100, over 1000) external data sources 8. These queries typically involve parameters such as time, location, or specific data types, necessitating rapid response times and high accuracy. The data queries are sent to the data management system, shown generally at 2. The data management system 2 receives the incoming queries at query ingestion subsystem 3A which provides the queries to data serving subsystem 3B that processes the queries. A data store 4, made up of a main memory and a fast-access cache memory, is queried by the data serving subsystem 3 to create results to the queries, and the results are then passed back by the data serving subsystem 3B to the end-user 1. The data in the data store 4 is sourced from potentially hundreds or thousands of external data sources 8 by a memory management subsystem 5 that makes calls or requests for data from these external data sources 8. The outgoing calls pass through an API 7 and the incoming data is processed by a data format reconciliation system 6 to ensure that a consistent data format is used for the data being passed to the data store 4 by the memory management subsystem 5. Conventional data processing systems face technical challenges in handling large-scale datasets that include dynamic, rapidly changing data. Typical technical challenges include: making the querying of the data store as computationally efficient as possible, minimising the time it takes to generate an accurate and comprehensive set of results for a query; making the system as flexible and extensible as possible so that it can readily cope with new query types and new data sources. SUMMARY OF THE INVENTION A first aspect of the invention is: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a query planner subsystem configured to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold. This approach enables all query combinations to be supported, and leads to faster and more computationally efficient querying of the data store and hence faster and more computationally efficient generation of accurate results that are fully responsive to the user query, even where the query is a complex one. Further details and aspects of the invention are given in the section headed Key Feature E. BRIEF DESCRIPTION OF THE FIGURES The invention will be described with reference to these Figures: Figure 1: is a schematic overview of a conventional data management system capable of processing thousands of user queries a second and generating results for those queries with data sourced from a very large number of external data sources. Figure 2 is a schematic overview of a data management system that shares the same purpose as the Figure 1 system, and that implements the invention. Figure 3 is a schematic overview of the query planner process. Figure 4 illustrates the identify stage of the query planner process. Figure 5 illustrates the ranking stage of the query planner process. Figure 6 and 7 are tables showing example file store scoring. Figure 8 is a table showing example file store scoring for a one-way London-New York in a single month example query. Figure 9 illustrates the building stage of the query planner process. Figure 10 illustrates the selection stage of the query planner process. Figure 11 illustrates the execution stage of the query planner process. DETAILED DESCRIPTION An implementation of this invention is used in the data management system shown schematically in Figure 2. Looking at Figure 2, as with a conventional data management system, multiple end-users 1 make data requests or user queries at their data terminals and these user queries are highly variable, often require accessing data originating from multiple external data sources 8. These user queries, as noted when we described Figure 1, typically involve parameters such as time, location, or specific data types, necessitating rapid response times and high accuracy. As before, the user queries are sent to the data management system, shown generally at 2. The data management system 2 receives the incoming user queries at query ingestion subsystem 3A which provides the user queries to data serving subsystem 3B that processes the user queries. A query planner subsystem 10 is responsible for determining the most efficient way to query the data store to generate accurate and comprehensive results to user queries. A data store 4, made up main memory and fast-access cache memory, stores the data that is processed to develop a result to a user query. Data store 4 is queried by the query planner subsystem 10 to create results to the user queries; we will describe this in detail later in this section. The results are then passed back by the data serving subsystem 3B to the end-user 1. Note that we are using the term 'query' in two different but related contexts: (a) the query the user formulates and (b) the query that the query planner makes to the memory. The context should make it clear which type of query we are referring to. As with the Figure 1 system, the data in the data store 4 is sourced from potentially thousands of external data sources 8 by a memory management subsystem 5 that makes calls or requests for data from these external data sources 8. The outgoing calls pass through an API 7 and the incoming data is processed by a data format reconciliation system 6 to ensure that a consistent data format is used for the data being passed to the data store 4 by the memory management subsystem 5. Data Store 4 includes a fast-access cache memory and a predictive database: The cache memory temporarily stores frequently accessed or recently updated data. The predictive database is used to predict data values for (some or all) of the same variables for which there is data in the cache memory. If there is a gap in the cache memory for some specific variables, then a user query can then be answered by using data from the predictive database instead; this eliminates the need for the query to be answerable only by a call to an external data source being made, waiting for the answer from the external data source, analysing the compatibility of that answer and then adding the answer to the data store. A trend analysis subsystem 11 analyses trends in the data in the predictive database and a gap detection subsystem 12 identifies gaps in the data held in the predictive database. A memory management subsystem 20 oversees the retrieval and updating of data from external datastores to the data store 4. Memory management subsystem 20 may include or connect to a cache memory management subsystem 13 that includes call management subsystem 14 that controls when and how calls for updated data to the cache memory are made, in order to minimize computational and network load while ensuring data accuracy. Cache memory management subsystem 13 also includes a monitoring subsystem 15 that monitors various call-related parameters, such as the call load - i.e. the number and / or frequency of calls or requests for updated data made to external data sources 8. The primary goal of the cache memory management subsystem 13 is to ensure that the data in the cache memory (and the main memory) is as up to date and accurate as possible, whilst ensuring overall system efficiency reliability and stability. This requires very careful management and organisation of the calls to the external data stores for refreshed data. As noted above, the query planner subsystem 10 is responsible for ensuring that user queries can be answered as quickly, and efficiently as possible. In the following section, we explain how the query planner subsystem 10 interacts with the data store 4 in an implementation of the invention. The Query Planner Subsystem In one implementation, the data storage querying system includes a query planner subsystem that determines the most efficient way to serve a given request or user query, and provides a response to that request quickly and efficiently. In the kinds of large-scale systems the invention is especially relevant to, there can be thousands of user queries a second, the queries require analysing very complex permutations of facts and an accurate response is required in seconds. The data storage querying system that implements the invention includes a memory that stores the data that is queried to provide a response to a user query. The memory is partitioned or organised into multiple datasets we call file stores. A file store organizes data in a way that allows fast retrieval using keys; file stores allow quick lookups based on predefined key structures. Multiple aggregations of the data may be pre-computed and written into different file-stores; the aggregations are each different permutations of the ingested data. The purpose of pre-computing these aggregations and storing these aggregations in the memory as multiple file stores is that the number of possible permutations relevant to a user query can be very large and performing large numbers of queries against a monolithic dataset will be very slow. By organising the data in the memory into separate file stores, each structured in a way that matches the organisation of one specific type or set of user queries, the memory is able to serve data using a look-up process that is responsive to user queries much more quickly and efficiently. An example: suppose the data relates to flights. Flight data has a very large number of key components; each key component is a parameter of the user query. Examples of key components are: origination airport; origination city; origination country; destination airport; destination city; destination country; day of flight; time of day of flight; airline; seat class; nonstop or multi-stop; price etc. Each user query will define some of these key components (e.g. a user could specify that they wish to see flights from London to Paris on 13th June, or they could input a broader search, such as flights from the UK to France in June). The possible potential combinations of these key components is very large. In one implementation, one file store could organise stored flight data on a per day basis, e.g. with pre-computed flight data sets for the following triplet combinations of data: {origination airport; destination airport; day}; {origination airport; destination city; day}; {origination city; destination airport; day}; {origination city; destination city; day}. Another file store could store pre-computed data sets organised on a per-month basis: {origination airport; destination airport; month}; {origination airport; destination city; month}; {origination city; destination airport; month}; {origination city; destination city; month}. Each of those triplet combinations of data is a key for a unique file store; the file store with the key {origination airport; destination airport; day} stores all of the flight data, organised as all origination airports and all destination airports reached by flights from each origination airports, for all days. Other file stores could organise the data into the following: a per destination country data set; a per destination city data set; a per origination city. To re-cap, a specific file store has a 'key', such as {origination airport; destination airport; day}. This key is made up of three 'key components', namely origination airport, destination airport and day. So we use the term 'key' to refer to a specific combination of key components. An example: LHR-CDG-2023-06-01 is a key that represents all the flights from London Heathrow (LHR) to Paris Charles de Gaulle (CDG) on June 1, 2023. The following tables list some of the file stores used in a flight search implementation: airport-airport-day airport-city-day city-airport-day city-city-day country-airport-day-ori gincity country-city-day-ori gincity country-airport-month-ori gincity country-city-month-ori gincity country-airport-ori gincity country-city-origincity airport-airport-month airport-city-month city-airport-month city-city-month country-airport-month country-city-month country-country-month city-country-month airport-country-month airport-day-country city-day-country country-day-country airport-month-country city-month-country country-month-country airport-country city-country country-country airport-country-day-destcity city-country-day-destcity country-country-day-destcity airport-country-month-destcity city-country-month-destcity country-country-month-destcity airport-country-destcity city-country-destcity country-country-destcity To understand why file stores organise the data in the way they do, note that the memory system can efficiently return data for a range of keys. This can be used to return data for a range of keys representing each day in a month. Key range queries rely on the keys in a file store being organised in a sequential order. To illustrate this concept, consider a simplified file store example containing flight quotes for a key made up of origin, destination and date. The three elements making up the key could be in any order but the order is crucial for query efficiency. If the order were origin-destination-date then this would enable efficient queries for each day in a month. For example, the key range LHR-CDG-2023-06-01 —> LHR-CDG-2023-06-30 would return all 30 quotes efficiently from a single or contiguous storage blocks for flights from London Heathrow airport (LHR) to Paris Charles de Gaulle (CDG). On the other hand, if the order of the key elements were different such as origin-date-destination, 30 separate keys would be required (LHR-2023-06-01-CDG, LHR-2023-06-02-CDG ... LHR-2023-06-30-CDG) making this query much more expensive. This second key would however be efficient for a query returning a price to every destination on a given date (e.g. LHR-2023-06-01-*). One implementation maintains 124 file stores (sometimes referred to as “skins”) in the memory, each with a key to support a specific query type. Each file store contains flight quotes organised by a composite key that is made up of a number of key components. There are 12 key components, each file store having a key made up of a number of these key components in different combinations. The table below lists the key components. originCountry originCity originAirport destinationcountry destinationCity destinationAirport outboundDepartureY earMonth outboundDepartureDay inboundDepartureY earMonth inboundDepartureDay marketingCarrier direct We will now describe the steps and related data storage querying systems that process an incoming query. In overview: The first step is the identification of which file stores are capable of answering the query. Once identified, the candidate file stores are scored and ranked. A query plan is then built for each identified file store starting with the highest ranked file stores and stopping when sufficient query plans have been built to make a decision on which is the optimal one. The optimal query plan is then built. Where no acceptable query plan has been identified, a message rejecting the request is returned to the client. The query plan is used to make a query to the memory using a FCP (file cache proxy) to obtain relevant results by informing the file store along with the key prefixes to be queried. The FCP is a general-purpose key-value data store and keys in a file store are organised in a sequential order to support key range queries. On receipt of the resulting quotes, local group-by aggregation is performed if necessary (depending on the query grouping option (if any) and which file store is used). Finally, the resulting quotes are returned to the client. We will now break this 'query planning' process down into the following key stages. Stage 1. Identify all file stores capable of answering a given query Stage 2. Rank all file stores capable of answering a given query Stage 3. Build the query plans. Stage 4. Select the optimal query plan. Stage 5. Execute by calling the file store and key range(s) defined by the identified optimal query plan to obtain quotes. Stage 6. In some cases, quotes loaded from the file stores may need to be reduced via aggregation. Figure 3 is a schematic overview of this entire six stage process. Looking at each stage in detail: Stage 1. Identify all file stores capable of answering a given query The first process is the identification of all file stores capable of answering a given query: The query planning subsystem takes each query, and identifies which file stores (if any) are suitable for serving that query. Example: London (city); Paris (city), June (month) Figure 4 shows schematically which file stores are capable of answering this query in this Identify phase. All file stores that are able to support the query are identified with a tick. At a high level, a check is made whether the file store supports the query resolution for each query parameter (return / one-way, origin geo level, destination geo level, outbound date resolution, inbound date resolution (if appropriate), carrier and grouping). By “supports” we mean the file store contains the necessary key components in its key. As an optimisation, the candidate file store identification may be precomputing the candidate file stores for each query combination. The set of suitable candidate file stores could be precomputed for query combination. Each of these query parameters are addressed in the following sections. One-way / Return The file store must support the one-way / retum query type. Origin Geo Level The file store must support at least the geo level of the query origin. • for airport level queries, the file store must include “originAirport” in its key • for city level queries, the file store must include either “originAirport” or “originCity” in its key • for country level queries, the file store must include either “originAirport”, “originCity” or “originCountry” in its key Note the assumption that, where there are multiple origin key components to the key, the keys will always appear in the key in a strict ordering of country, city, airport from left to right. Destination Geo Level As for origin geo level but for destination Airport, City and Country. Destination Geo Level Detail: The file store must support at least the geo level of the query destinations (all destinations must be at the same level). • for airport level queries, the file store must include “destinationAirporf ’ in its key • for city level queries, the file store must include either “destinationAirporf’ or “destinationCity” in its key • for country level queries, the file store must include either “destinationAirporf’, “destinationCity” or “destinationcountry” in its key Note the assumption that, where there are multiple origin key components to the key, the keys will always appear in the key in a strict ordering of country, city, airport from left to right. Outbound Date Resolution The file store must support at least the query outbound resolution. • for day resolution queries, the file store must include “outboundDepartureDay” in its key • for month resolution queries, the file store must include either “outboundDepartureDay” or “outboundDepartureYearMonth” in its key Note the assumption that “outboundDepartureYearMonth” must be present in the key where “outboundDepartureDay” is present and month must be to the left of the day in the key order. The file store must support at least the query inbound resolution. Inbound is only applicable to return queries. For day resolution queries, the file store must include “inboundDepartureDay” in its key. For month resolution queries, the file store must include either “inboundDepartureDay” or “inboundDepartureYearMonth” in its key. Note the assumption that “outboundDepartureYearMonth” must be present in the key where “outboundDepartureDay” is present and month must be to the left of the day in the key order. Inbound Date Resolution As for outbound date resolution but for inbound DepartureDay and DepartureYearMonth. Example Output The following lists all candidate file stores for a one-way London-New York, single month, per destination example query. oneway originAirport destinationAirport outboundDepartureYearMonth Direct oneway originAirport destinationCity outboundDepartureYearMonth Direct oneway_originCity destinationAirport outboundDepartureYearMonth Direct oneway originCity destinationCityoutboundDepartureYearMonthDirect oneway _originAirport_destinationCountry_outboundDepartureYearMonth_destinationCity_ Direct oneway originCity destinationcountry outboundDepartureYearMonth destinationCity Dir ect oneway originAirport destinationAirport outboundDepartureYearMonth outboundDepartur eDay Direct oneway originAirport destinationCity outboundDepartureYearMonth outboundDepartureD ay Direct oneway_originCity destinationAirport outboundDepartureYearMonth outboundDepartureD ay Direct oneway originCity destinationCity outboundDepartureYearMonth outboundDepartureDay Direct oneway_originCountry destinationAirport outboundDepartureYearMonth outboundDepartu reDay originCity Direct oneway originCountry destinationCity outboundDepartureYearMonth outboundDeparture Day originCity Direct oneway_originCountry destinationAirport outboundDepartureYearMonth originCity Direct oneway originCountry destinationCity outboundDepartureYearMonth originCity Direct oneway originAirport destinationCountry outboundDepartureYearMonth outboundDepartu reDay destinationCity Direct oneway originCity destinationcountry outboundDepartureYearMonth outboundDeparture Day destinationCity Direct Stage 2. Rank all file stores capable of answering a given query Once the candidate file stores have been identified, query plans need to be built for them and used to determine the optimal query plan. Before the query plans are built, the candidate file stores are scored and ranked so that the most appropriate candidates can be considered first and query plan construction can finish early when a query plan has been built and the next candidate has a higher score. This enables multiple candidates with the same score to be considered together. A score from 0 to 1 is assigned to each possible key component. A score of 0 means that the key component does not support the query and 1 means that the key component is a perfect match for the query (e.g. originCity when the query origin is a city). A score between 0 and 1 is assigned for an imperfect match (e.g. originAirport when the query origin is a city). These scores may be weighted differently should we discover the ranking algorithm is not operating optimally. Any file store with a score of 0 for any key component should have been discarded before reaching this phase and so zero scores should not be apportioned here. As an optimisation, the suitability score could be precomputed for each file store for each query combination. Figure 5 schematically shows this Rank phase. Destination Scoring The Figure 6 table shows scores assigned for origins and destinations. Omitted combinations score zero. The Figure 7 table shows scores assigned for outbound and inbound dates. Omitted combinations score zero. The difference of the score for each key component from 1 is summed to give a sum of differences which is used to group file stores into similar suitability groups and to rank the likely best matches. A simple sort of these values yields a rank. Example Output The Figure 8 table shows the scores for a one-way London-New York in a single month example query. We can see there are three file stores with the same perfect 1 score and if an acceptable query plan can be built from one of these then there is no need to build any more query plans as the optimal from these three can be used. Stage 3. Build the query plans. Once the candidate file stores have been identified and ranked, they can be considered for its suitability to serve the query. The system starts with the top ranked file store, and then builds key trees for all file stores with the same rank, and stops for any key tree that becomes too wide, and stops when a rank has a viable plan to answer the query. A tree of key values is built covering each component of the key at each level. The tree informs the maximum number of results (assuming full coverage) that would be returned by the cache or FCP (the number of leaves in the tree) and the number of key prefixes to be requested from the cache or FCP (the width of the tree at the level beyond which all keys are required). These are useful metrics for comparing the cost of each query. The tree also enables identification of the key prefixes to be sent to the cache or FCP. Key tree construction can be expensive if trees are constructed for every candidate file store. This is because the number of subtrees can become very large. An extreme example may be an “anywhere” query using an airport-level file store. In this case, the tree will split into several thousand subtrees - one for each airport in the world. This query would need to be rejected due to its size and therefore it would not be worthwhile building a query plan of such size. To counteract these inefficiencies, a number of optimisations are applied at this phase. Firstly, the file stores are ordered processed in groups of file stores with the same rank. Secondly, key trees are built breadth-first rather than depth-first so that the width of the tree can be monitored during construction and the process bailed-out of if the tree becomes too wide. Once an acceptable plan has been identified the query plan generation can be stopped before proceeding to the next rank of file stores. Figure 9 schematically shows this Build phase. Stage 4. Select the optimal query plan. The system selects the optimal plan using: (i) Estimated block count; (ii) Number of key ranges (iii) Result count; (iv) Number of key components. Example Query Plans The example query plans continue the example from previous sections (London to New York, one-way, departing 2023-06 with the default (per destination rollup). Query plans are grouped by their rank, each tree representing a different query plan. The example query plans go further than the processing would go - this would stop after building the first 3 plans for rank 1=. Each query plan is ranked using the following properties in the following order: 1. Estimated number of blocks to be read (via FCP) 2. Number of key prefixes to be requested from FCP 3. Total number of results to be returned (number of leaves in the tree / maximum tree width) 4. Number of key components (tree height) Trees are ranked using a function of the number of key prefixes and the number of returned results with the tree height being used as a tie-breaker. The comparison uses the prefix count and the number of results returned to estimate the number of blocks to be accessed by FCP. The number of prefixes is multiplied by the estimated number of blocks per prefix. The number of blocks per prefix is calculated by dividing the number of results by the number of prefixes (subject to a maximum number of keys per block). The above example shows the scores for each plan in red (assuming a MAXKEYSINBLOCK value of 30). MAXKEYSINBLOCK will be determined via analysis of the existing data. Should no acceptable plan be identified, an API returns a response to the client rejecting the query. The cost of each query is considered so that queries can be rejected if deemed too expensive -this is achieved by calculating the number of key prefixes that need to be requested from the cache or FCP and the total number of results that may be returned (assuming full coverage). The cost of each query is logged so that expensive queries can be considered as candidates for optimisation (for example by adding new file stores) or conversations with clients held to optimise usage. The Query Planner is configured with a list of available file stores, the addition of a new file store only requiring a configuration change so that the Query Planner will consider it for queries. Figure 10 schematically shows this Select phase. Stage 5. Execute by calling the FCP with the file store and key range(s) defined by the identified optimal query plan to obtain quotes. After the optimal query plan has been determined, an API calls the FCP in a single call with the list of key prefixes identified in the plan. The FCP returns all matching quotes. The FCP is always called with discrete key ranges (no wildcards). These is a maximum of 1 call per client request. Figure 11 schematically shows this Execute phase. Stage 6. In some cases, quotes loaded from cache may need to be reduced via aggregation. This happens when a sub-optimal file store needs to be used to satisfy a query. Examples might be: an "anytime” query needs to obtain a quote for each of the 12 months ahead and find the cheapest; a "month" query needs to use a day level file store and select the cheapest quote from each day in the month. Occurrences are an opportunity for detecting whether a new file store is required. Once matching quotes have been returned by FCP, the final stage is to perform group-by aggregation grouping the quotes as defined by the request grouping option (for example by destination, or month) and selecting the quote with the min price (both direct and indirect where indirect is cheaper). Where the “perfect” file store has been selected, this will be a no-op but, in some cases, where the perfect file store does not exist for a given query a group-by aggregation needs to be performed. For example, a query for the cheapest quote to each destination city in a country might need to group the quotes by destination city, choosing the cheapest for each. This process implies an element of waste because some quotes returned by FCP will be discarded. Such waste should be avoided where possible by selection of the most appropriate file store. We will now describe significant features of the data management system that implements the invention; we refer to them as 'Key Features', to distinguish them from the conventional. These 'Key Features' are fundamental to how the overall data management system for processing user queries is able to serve so many users, so quickly and reliably, with data that is constantly altering from potentially 1000s of different external partners. Note that any of the high-level ‘Key Features’ can be combined with one or more of the other features, and any of the optional features. Any of the optional features can be combined with one or more of the other optional features. Key Feature A. File store structure optimised for query response Conventional data management systems can improve query performance using dynamic indexing strategies that do not modify the underlying data structures. But dynamic indexing has limitations. Challenges in these conventional systems include: • Increased computational overhead during query execution due to reliance on dynamic computations. • Inefficiencies in handling diverse and high-volume query patterns with complex parameter combinations To address these limitations, in Key Feature A the data store is organised into multiple file stores, each organised for specific, ordered query pattern combinations. For example, where the data management system is used to answer flight search queries, then the specific query pattern combination could include for one file store data organised in this combination: originating airport - destination airport - day. Another file store could use a different combination: originating city - destination city - month. These different file stores will be relevant to different search queries: the first file store would be relevant to a flight search query such as 'flights from London Heathrow to Paris Charles de Gaulle on 16 May', and the second file store would be relevant to a flight search query such as 'flights from any UK airport to any French airport in May'. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys. Using the example above, the 'key' for one file store is the combination {originating airport -destination airport - day} and there are three key components: (1) {originating airport}; (2) { destination airport} and (3) {day}. The key for the other file store is {originating city -destination city - month} and there are three key components: (1) {originating city}; (2) {destination city} and (3) {month}. Some implementation features: • Domain-Specific File Store Organization: Each file store is pre-structured based on anticipated query patterns, such as combinations of origin, destination, and time. • Reduced Query Latency: By organizing data at the file store level, the system minimizes on-the-fly computations, improving query response times. • Improved Resource Utilization: The tailored organization of data reduces computational overhead, optimizing resource usage for high-volume query environments involving complex combinations of query terms. Unlike traditional indexing systems that rely on dynamic computations during query execution, this system structures the data itself to optimize retrieval efficiency for domain-specific, complex queries. This granular organization is uniquely structured for the domain - i.e. each file store is organised into a specific query pattern that matches or corresponds to different potential combinations of parameters (described as 'key components') in a user query. The data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys. It also introduces flexibility and extensibility: new file stores, with new combinations of parameters or 'key components' can be readily added and that is all that is needed - no complex code changes is needed. This approach enables the system to precompute and store datasets organized by granularities or combinations of parameters such as origin, destination, and time (e.g., day, month). By organizing the data into tailored file stores, the system reduces the need for on-the-fly computations during query execution, significantly improving response times. This systematic partitioning introduces a level of granularity to data organization, ensuring that query patterns specific to the domain are addressed with minimal computational overhead. This approach allows the data to be pre-structured for domain-specific queries, reducing search complexity and improving the system's overall efficiency. By tailoring file stores to specific query patterns, the system optimizes query execution processes, reducing latency and computational overhead. The introduction of multiple file stores within the data store architecture enhances query handling efficiency. Examples of use cases are now provided. Implementing this Key Feature A improves the technical operation of these systems by adapting a data processing system to domain-specific key patterns, enabling faster and more efficient data retrieval. Some use cases: • Travel booking (e.g. flight, hotel or car hire search) platform: file stores are organized to store data organised to ensuring rapid access to flight / hotel / car availability and pricing information. • Financial data systems: file stores are optimized for asset classes or trading hours to enable fast retrieval of market data. • Environmental data systems: file stores may be prestructured for combinations of geographic regions and temporal parameters, allowing rapid access to climate data or geological surveys. • Al-powered robotics: file stores can be organized based on sensor types and operational zones, enabling real-time access to critical navigation or environmental information. We can also generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys. And we can generalise to: A method for improving the processing and querying of data in a data management system, comprising: (a) structuring a data store for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and in which the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys. Key Feature B. Efficient identification of relevant file stores Identifying applicable data sources or indexes for a query is normal in query planners. Most relational or distributed databases have metadata systems to track the suitability of file partitions or indexes for specific queries. Conventional query planners utilize static metadata or predefined indexes to determine the suitability of data partitions or indexes for executing queries. While effective in predictable and static environments, conventional systems face challenges in scenarios involving: • Query patterns that can rapidly alter. For example, in the flight or hotel search context, new user selectable search variables are frequently introduced to make these systems ever more responsive to user's querying needs. For example, in the hotel search context, new search parameters could be introduced, like 'has direct beach access' or 'great night life' or 'family friendly'. This can require re-coding, which can be slow and introduce bugs or lead to other unexpected consequences. • Complex, high-volume query loads with very complex parameter combinations. These limitations lead to inefficiencies in resource utilization and query execution, as static approaches fail to adapt to real-time changes in query requirements. To address these limitations, Key Feature B implements the file store structure described in Key Feature A, and introduces a file store identification subsystem configured to identify file stores that are capable of answering a specific query by analysing the key components for a number of file stores. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a file store identification subsystem configured to identify file stores that are capable of answering a specific query by analysing the key components for a number of file stores. This is a computationally fast and efficient way of identifying file stores that are capable of answering a specific query, especially where the search domain involves a potentially large number of permutations of search parameters (especially true of flight, hotel and car hire search). We can also generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and there is (vii) a file store identification subsystem configured to identify file stores that are capable of answering a specific query by analysing the key components for a number of file stores. And we can generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a file store identification subsystem to identify file stores that are capable of answering a specific query by analysing the key components for a number of file stores. Key Feature C. Efficient ranking of all file stores capable of answering a given query Ranking potential query paths based on factors like specificity and efficiency is a normal feature in query planners. Conventional query planners rank potential query paths based on static factors such as specificity, computational cost, or metadata analysis. These methods, while effective in general scenarios, suffer from limitations: • Inefficiencies in ranking file stores without directly accessing partition maps or data content. • Higher computational overhead and latency in environments with high query volumes and complex data structures with a large number of complex query combinations. Existing query planners lack the ability to incorporate domain-specific optimisation into their ranking processes, leading to suboptimal performance in data-intensive applications. To address these technical limitations, Key Feature C implements the file store structure described in Key Feature A, and introduces a ranking subsystem adapted to score each key component for each file store for relevance or match to a specific query and to use the scores to rank the file stores. This is a fast, efficient, flexible and scalable approach. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a ranking subsystem adapted to score each key component for each file store for relevance or match to a specific query and to use the scores to rank the file stores. The data storage querying system uses multiple unique file stores that are each organised in a manner that matches the possible query combinations (Key Feature A above) as defined by a unique key for each file store, and the ranking system leverage this to use a simple and fast relevance scoring mechanism, based on scoring each key component in the key for relevance to a specific query. An example: if the key for file store A is {originating airport - destination airport - day} and the key for file store B is {originating city - destination city - month}, and the query is 'find me flights from London Heathrow to Paris Charles de Gaulle on 16 May', then file store A will be ranked higher than store B - file store A is in fact a perfect match since all of its individual key components, i.e. originating airport, destination airport, day are fully relevant to the search query. In more detail: once the candidate file stores have been identified, query plans need to be built for them and used to determine the optimal query plan. Before the query plans are built, the candidate file stores are scored and ranked so that the most appropriate candidates can be considered first and query plan construction can finish early when a query plan has been built and the next candidate has a higher score. This enables multiple candidates with the same score to be considered together. We can also generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and there is: (vii) a ranking subsystem adapted to score each key component for each file store for relevance or match to a specific query and to use the scores to rank the file stores. And we can generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a ranking subsystem to score each key component for each file store for relevance or match to a specific query and to use the scores to rank the file stores. Key Feature D. Efficient identification of file stores using key trees Key Feature D implements the file store structure described in Key Feature A, and introduces a query planner subsystem configured to (i) rank each file store for relevance to a specific query and (ii) to identify viable file stores that could respond to the query by starting with each top ranked file store, and then building key trees for all file stores grouped within the top rank, each tree representing a different query plan. Using key trees is a fast and efficient way to profile different file stores. The data storage querying system starts with the top ranked file store, and then builds key trees for all file stores with the same rank, and stops for any key tree that becomes too wide, and stops when a rank has a viable plan to answer the query. A tree of key values is built covering each component of the key at each level. The tree informs the maximum number of results (assuming full coverage) that would be returned by the FCP (the number of leaves in the tree) and the number of key prefixes to be requested from the FCP (the width of the tree at the level beyond which all keys are required). These are useful metrics for comparing the cost of each query. The tree also enables identification of the key prefixes to be sent to the FCP. Key tree construction can be expensive if trees are constructed for every candidate file store. This is because the number of subtrees can become very large. An extreme example may be an “anywhere” query using an airport-level file store. In this case, the tree will split into several thousand subtrees - one for each airport in the world. This query would need to be rejected due to its size and therefore it would not be worthwhile building a query plan of such size. To counteract these inefficiencies, a number of optimisations may be applied at this phase. Firstly, the file stores can be ordered processed in groups of file stores with the same rank. Secondly, key trees can be built breadth-first rather than depth-first so that the width of the tree can be monitored during construction and the process bailed-out of if the tree becomes too wide. Once an acceptable plan has been identified the query plan generation can be stopped before proceeding to the next rank of file stores. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a query planner subsystem configured to (i) rank each file store for relevance to a specific query and (ii) to identify viable file stores of each rank that could respond to the query, by starting with each top ranked file store, and then building key trees for all file stores grouped within the top rank, each tree representing a different query plan, and if none of the file stores at that top rank are viable, key trees are then constructed for the file stores of the next rank. And to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and the query planner subsystem is configured to (i) rank each file store for relevance to a specific query and (ii) to identify viable file stores of each rank that could respond to the query, by starting with each top ranked file store, and then building key trees for all file stores grouped within the top rank, each tree representing a different query plan, and if none of the file stores at that top rank are viable, key trees are then constructed for the file stores of the next rank. And we can generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a query planner subsystem to (i) rank each file store for relevance to a specific query and to (ii) identify viable file stores of each rank that could respond to the query, by starting with each top ranked file store, and then building key trees for all file stores grouped within the top rank, each tree representing a different query plan, and if none of the file stores at that top rank are viable, constructing key trees for the file stores of the next rank. Key Feature E. Stopping overload by analysing tree width As noted above, one technical advantage of analysing file stores by building key trees for different candidate file stores is that it gives a clear indication of the computational cost of using that file store to support or execute a specific query plan. But key tree construction can be expensive if trees are constructed for every candidate file store. This is because the number of subtrees can become very large. An extreme example may be an “anywhere” query using an airport-level file store. In this case, the tree will split into several thousand subtrees - one for each airport in the world. This query would need to be rejected due to its size and therefore it would not be worthwhile building a query plan of such size. Key Feature E address this; implements the file store structure described in Key Feature A, and introduces a query planner subsystem configured to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold. In more detail: Firstly, the file stores are ordered processed in groups of file stores with the same rank. Secondly, key trees are built breadth-first rather than depth-first so that the width of the tree can be monitored during construction and the process bailed-out of if the tree becomes too wide. Once an acceptable plan has been identified the query plan generation can be stopped before proceeding to the next rank of file stores. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a query planner subsystem configured to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold. And we can generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and the query planner subsystem is configured to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold. And we can also generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a query planner subsystem to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold. Key Feature F: Selecting the optimal query plan Key Feature F implements the file store structure described in Key Feature A, and defines how the query planner subsystem selects the optimal query plan in a very efficient and fast manner. Each query plan is ranked using the following properties in the following order: 1. Estimated number of blocks to be read (via FCP) 2. Number of key prefixes to be requested from FCP 3. Total number of results to be returned (number of leaves in the tree / maximum tree width) 4. Number of key components (tree height) Trees are ranked using a function of the number of key prefixes and the number of returned results with the tree height being used as a tie-breaker. The comparison uses the prefix count and the number of results returned to estimate the number of blocks to be accessed by FCP. The number of prefixes is multiplied by the estimated number of blocks per prefix. The number of blocks per prefix is calculated by dividing the number of results by the number of prefixes (subject to a maximum number of keys per block). Should no acceptable plan be identified, an API returns a response to the client rejecting the query. The cost of each query is considered so that queries can be rejected if deemed too expensive -this is achieved by calculating the number of key prefixes that need to be requested from the cache or FCP and the total number of results that may be returned (assuming full coverage). The cost of each query is logged so that expensive queries can be considered as candidates for optimisation (for example by adding new file stores) or conversations with clients held to optimise usage. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a query planner subsystem configured to (i) identify file stores with a viable query plan for responding to a query, and to build key trees for one or more viable query plans and in which the query planner subsystem selects the optimal query plan by identifying the file store that results in the smallest number of blocks being loaded from backing storage. We can also generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and the query planner subsystem is configured to (i) identify file stores with a viable query plan for responding to a query, and to build key trees for one or more viable query plans and in which the query planner subsystem selects the optimal query plan by identifying the file store that results in the smallest number of blocks being loaded from backing storage. And we can generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a query planner subsystem to (i) identify file stores with a viable query plan for responding to a query, and to build key trees for one or more viable query plans and in which the query planner subsystem selects the optimal query plan by identifying the file store that results in the smallest number of blocks being loaded from backing storage. Key Feature G: Adding new file stores Key Feature G implements the file store structure described in Key Feature A, and defines how the query planner subsystem can identify where new file stores can be added to more efficiently support specific queries that existing file stores are not optimised for. The query planner subsystem is configured with a list of available file stores; the addition of a new file store only requires a configuration change so that the query planner subsystem will consider it for queries. This makes the data storage querying system rapidly reconfigurable and flexible, without requiring any new coding. The cost of each query is considered so that queries can be rejected if deemed too expensive -this is achieved by calculating the number of key prefixes that need to be requested from the cache or FCP and the total number of results that may be returned (assuming full coverage). The cost of each query is logged so that expensive queries can be considered as candidates for optimisation by adding new file stores that are organised specifically to support those new queries (e.g. have key combinations that match the structure of the otherwise expensive queries. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a query planner subsystem configured to (i) identify file stores with a viable query plan for responding to a query, and to (ii) cost an optimal viable query plan by estimating or calculating the number of key prefixes or ranges that need to be requested from the data store and (iii) identifying expensive queries that exceed a cost threshold as candidates for optimisation by adding new file stores that more efficiently support those queries. We can also generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and the query planner subsystem is configured to (i) identify file stores with a viable query plan for responding to a query, and to (ii) cost an optimal viable query plan by estimating or calculating the number of key prefixes or ranges that need to be requested from the data store and (iii) identifying expensive queries that exceed a cost threshold as candidates for optimisation by adding new file stores that more efficiently support those queries. And we can generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a query planner subsystem to (i) identify file stores with a viable query plan for responding to a query, and to (ii) cost an optimal viable query plan by estimating or calculating the number of key prefixes or ranges that need to be requested from the data store and (iii) identifying expensive queries that exceed a cost threshold as candidates for optimisation by adding new file stores that more efficiently support those queries. Key Feature H. Execute with a single call Conventional systems may need to make multiple calls for a single query, leading to storage layer fan-out and increased computational overhead. These inefficiencies create challenges in environments with high query volumes or complex data structures, resulting in degraded performance, increased latency, and scalability issues. This Key Feature H stipulates that the data storage querying system includes a query planner mechanism that enforces a strict optimization constraint: all queries must be executed using a single, specific call, avoiding reliance on wildcard operations. Unlike conventional systems that optimize key ranges but still allow broad wildcard queries or multiple calls, this mechanism ensures that each query targets a precise set of data keys, minimizing computational overhead and storage layer fan-out. By enforcing a single-call-per-query constraint, the system reduces latency, simplifies integration, and ensures predictable and scalable query execution. The query planner eliminates wildcard operations entirely, refining query plans to retrieve only relevant data. This prevents the inefficiencies and performance degradation caused by unnecessary data retrieval and excessive storage layer requests. This approach optimizes resource usage, improves scalability, and prevents fan-out to the storage layer. By integrating a wildcard elimination mechanism and a fan-out prevention subsystem, the system dynamically refines query execution paths, ensuring compliance with the single-call constraint. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a query planner subsystem configured to (i) identify file stores with a viable query plan for responding to a query; (ii) generate an optimal query plan; and (iii) execute the optimal plan using a single call to the data store. The system is designed to execute each query using a single, specific call that avoids wildcard operations entirely. This constraint simplifies integration, improves predictability in cache behavior, and ensures that the storage layer is not subjected to the high load and complexity associated with fan-out patterns. By focusing each call on a precise set of data keys, the system minimizes the computational resources required for query execution and avoids the unnecessary retrieval of irrelevant data. This strict optimization rule provides significant technical advantages by reducing query latency, improving API efficiency, and preventing the performance degradation caused by fanout to the storage layer. Additionally, by avoiding wildcards, the system ensures tighter control over query execution, making it easier to scale and maintain in complex, data-intensive environments. The system is able to enforce a simplified and highly efficient query execution strategy. By constraining queries to a single call, the system achieves predictable performance improvements and avoids the pitfalls of traditional multi-call or wildcard-based approaches, offering a practical solution to a common challenge in cache-based systems. We can also generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; the query planner subsystem is configured to (i) identify file stores with a viable query plan for responding to a query; (ii) generate an optimal query plan; and (iii) execute the optimal plan using a single call to the data store. And we can generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a query planner subsystem to (i) identify file stores with a viable query plan for responding to a query; (ii) generate an optimal query plan; and (iii) execute the optimal plan using a single call to the data store. Key Feature I. Aggregating results In some cases, quotes loaded from cache may need to be reduced via aggregation. This happens when a sub-optimal file store needs to be used to satisfy a query. Examples might be: an ’’anytime” query needs to obtain a quote for each of the 12 months ahead and find the cheapest; a month query needs to use a day level file store and select the cheapest quote from each day in the month. Occurrences are an opportunity for detecting whether a new file store is required. Once matching results have been returned by FCP, the final stage is to perform group-by aggregation grouping the quotes as defined by the request grouping option (for example by destination, or month) and selecting the quote with the min price (both direct and indirect where indirect is cheaper). Where the “perfect” file store has been selected, this will be a no-op but, in some cases, where the perfect file store does not exist for a given query a group-by aggregation needs to be performed. For example, a query for the cheapest quote to each destination city in a country might need to group the quotes by destination city, choosing the cheapest for each. This process implies an element of waste because some quotes returned by FCP will be discarded. Such waste should be avoided where possible by selection of the most appropriate file store. We can generalise to: A data storage querying system, comprising: (a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and (b) a query planner subsystem configured to (i) identify file stores with a viable query plan for responding to a query; (ii) generate an optimal query plan; (iii) execute the optimal plan and receive matching results back; and (iv) aggregate matching results to return results requested in the query, where the query to the data store was not a perfect match. We can also generalise to: A data management system configured for generating results for user queries, the system including: (i) a query ingestion subsystem configured to receive end-user queries; (ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query; (iii) a data store storing data that is queried by the query planner; (iv) a call management system that controls requests to external data sources; (v) an API that provides an interface to the external data sources; (vi) a data serving system that receives and processes queries from end-users and serves results to end-users; in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; the query planner subsystem configured to (i) identify file stores with a viable query plan for responding to a query; (ii) generate an optimal query plan; (iii) execute the optimal plan and receive matching results back; and (iv) aggregate matching results to return results requested in the query, where the query to the data store was not a perfect match. And we can generalise to: A method for improving the processing and querying of a data store, comprising: (a) structuring a data store for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; (b) operating a query planner subsystem to (i) identify file stores with a viable query plan for responding to a query; (ii) generate an optimal query plan; (iii) execute the optimal plan and receive matching results back; and (iv) aggregate matching results to return results requested in the query, where the query to the data store was not a perfect match. For all of the preceding Key Features, one or mor of the following features may also be relevant: FCP • the data store is configured to enable the fast retrieval of data using a file cache proxy or FCP. • the FCP is a general-purpose key-value data store. • the file stores are maintained in the FCP. • keys in a file store are organised in a sequential order to support key range queries. Key components • the system includes a query input subsystem configured to organise a user query into a number of key components that each define a parameter or variable for that query. • each key component defines a parameter or variable relevant to categorising a user query. • key components are organised into hierarchies. The Query Planner • a query planner subsystem is configured to decide which file stores (if any) are suitable for serving a query, choose the optimal file store and provide the key ranges to be sent in a request toFCP. • the query planner subsystem is configured with a list of available file stores, and the addition of a new file store only requires a configuration change so that the query planner subsystem will consider it for queries. • a file store identification subsystem is configured to identify file stores that are capable of answering a specific query by analysing the key components for each file store. Query Planner: Ranking and Building • a ranking subsystem is adapted to score each key component for each file store for relevance or match to the specific query and to use the scores to rank the file stores. • the query planner subsystem is configured to plan a query and to start with top ranked file stores, and then build key trees for all file stores grouped with the same rank, each tree representing a different query plan. o the query planner subsystem is configured to stop for any key tree that becomes too wide. o key trees are built breadth-first rather than depth-first so that the width of the tree can be monitored during construction and the process bailed-out of if the tree becomes too wide. o the tree informs the maximum number of results (assuming full coverage) that would be returned by the FCP (the number of leaves in the tree) and the number of key prefixes to be requested from the FCP (the width of the tree at the level beyond which all keys are required). o the tree enables identification of the key prefixes or ranges to be sent to the FCP. o the query planner subsystem is configured to identify all file stores of a given rank that have a viable query plan to answer the query. o the query planner subsystem is configured so that once an acceptable plan has been identified the query plan generation can be stopped before proceeding to the next rank of file stores. Query Planner: Selection • a query planner subsystem is configured to select an optimal query plan o the query planner subsystem selects the optimal plan by analysing for each query plan: (i) Estimated block count; (ii) Number of key ranges (iii) Result count (e.g. number of leaves in the tree / maximum tree width); (iv) Number of key components (e.g. (tree height). o trees are ranked using a function of the number of key prefixes and the number of returned results, with the tree height being used as a tie-breaker. o the ranking uses the prefix count and the number of results returned to estimate the number of blocks to be accessed by FCP, and the number of prefixes is multiplied by the estimated number of blocks per prefix, and the number of blocks per prefix is calculated by dividing the number of results by the number of prefixes. o the cost of each query is analysed so that queries can be rejected if deemed too expensive. o query cost analysis is achieved by calculating the number of key prefixes that need to be requested from the FCP and the total number of results that may be returned, assuming full coverage. o the cost of each query is logged so that expensive queries can be considered as candidates for optimisation, for example by adding new file stores whose cost will be lower. Query Planner: Execution • the query planner subsystem is configured to execute the optimal plan using a single call to the data store. • the data required to respond to a query' is contiguously located in the file-store so that it can be returned efficiently using a single call to the FCP. • the data required to respond to a query' is contiguously located in the file-store in a single key range so that it can be returned efficiently in a single call to the FCP. • after the optimal query plan has been determined, a single call to the data store via FCP is made with the list of key prefixes or ranges identified in the optimal plan, and the FCP returns all matching results. • the FCP is always called with discrete key ranges and no wildcards. • there is a maximum of 1 FCP call per client request. Query Planner: Aggregation • the query planner subsystem is configured to aggregate matching results to reduce the number of results, before presenting the results to the user o Once matching quotes have been returned by FCP, the final stage is to perform group-by aggregation, grouping the results as defined by the request grouping option (for example by destination, or month) and ranking the results, e.g. leading with a result with the min price • the query planner subsystem is configured to identify when a new file store is needed to avoid the need to aggregate matching quotes or results.

Claims

1. A data storage querying system, comprising:(a) a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys; and(b) a query planner subsystem configured to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold.FCP2. The data storage querying system of Claim 1 in which the data store is configured to enable the fast retrieval of data using a file cache proxy or FCP.

3. The data storage querying system of Claim 2 in which the FCP is a general-purpose key-value data store.

4. The data storage querying system of Claim 3 in which the file stores are maintained in the FCP.

5. The data storage querying system of any preceding Claim in which keys in a file store are organised in a sequential order to support key range queries.Key components6. The data storage querying system of any preceding Claim in which the system includes a query input subsystem configured to organise a user query into a number of key components that each define a parameter or variable for that query.

7. The data storage querying system of any preceding Claim in which each key component defines a parameter or variable relevant to categorising a user query.

8. The data storage querying system of any preceding Claim in which key components are organised into hierarchies.The Query Planner9. The data storage querying system of any preceding Claim in which a query planner subsystem is configured to decide which file stores (if any) are suitable for serving a query, choose the optimal file store and provide the key ranges to be sent in a request to FCP.

10. The data storage querying system of any preceding Claim in which the query planner subsystem is configured with a list of available file stores, and the addition of a new file store only requires a configuration change so that the query planner subsystem will consider it for queries.

11. The data storage querying system of any preceding Claim in which a file store identification subsystem is configured to identify file stores that are capable of answering a specific query by analysing the key components for each file store.Query Planner: Ranking and Building12. The data storage querying system of any preceding Claim in which a ranking subsystem is adapted to score each key component for each file store for relevance or match to the specific query and to use the scores to rank the file stores.

13. The data storage querying system of any preceding Claim in which a query planner subsystem is configured to plan a query and to start with top ranked file stores, and then build key trees for all file stores grouped with the same rank, each tree representing a different query plan.

14. The data storage querying system of preceding Claim 13 in which the query planner subsystem is configured to stop for any key tree that becomes too wide.

15. The data storage querying system of preceding Claim 14 in which key trees are built breadth-first rather than depth-first so that the width of the tree can be monitored during construction and the process bailed-out of if the tree becomes too wide.

16. The data storage querying system of any preceding Claim 13 - 15 in which the tree informs the maximum number of results (assuming full coverage) that would be returned by the FCP (the number of leaves in the tree) and the number of key prefixes to be requested from the FCP (the width of the tree at the level beyond which all keys are required).

17. The data storage querying system of any preceding Claim 13 - 16 in which the tree enables identification of the key prefixes or ranges to be sent to the FCP.

18. The data storage querying system of any preceding Claim in which a query planner subsystem is configured to identify all file stores of a given rank that have a viable query plan to answer the query.

19. The data storage querying system of any preceding Claim in which a query planner subsystem is configured so that once an acceptable plan has been identified the query plan generation can be stopped before proceeding to the next rank of file stores.Query Planner: Selection20. The data storage querying system of any preceding Claim in which a query planner subsystem is configured to select an optimal query plan.

21. The data storage querying system of preceding Claim 20 in which the query planner subsystem selects the optimal plan by analysing for each query plan: (i) Estimated block count; (ii) Number of key ranges (iii) Result count (e.g. number of leaves in the tree / maximum tree width); (iv) Number of key components (e.g. (tree height).

22. The data storage querying system of preceding Claim 21 in which trees are ranked using a function of the number of key prefixes and the number of returned results, with the tree height being used as a tie-breaker.

23. The data storage querying system of preceding Claim 22 in which the ranking uses the prefix count and the number of results returned to estimate the number of blocks to be accessed by FCP, and the number of prefixes is multiplied by the estimated number of blocks per prefix, and the number of blocks per prefix is calculated by dividing the number of results by the number of prefixes.

24. The data storage querying system of any preceding Claim 21 - 23 in which the cost of each query is analysed so that queries can be rejected if deemed too expensive.

25. The data storage querying system of preceding Claim 24 in which query cost analysis is achieved by calculating the number of key prefixes that need to be requested from the FCP and the total number of results that may be returned, assuming full coverage.

26. The data storage querying system of any preceding Claim 24 - 25 in which the cost of each query is logged so that expensive queries can be considered as candidates for optimisation, for example by adding new file stores whose cost will be lower.Query Planner: Execution27. The data storage querying system of any preceding Claim in which a query planner subsystem is configured to execute the optimal plan using a single call to the data store.

28. The data storage querying system of preceding Claim 27 in which the data required to respond to a query is contiguously located in the file-store so that it can be returned efficiently using a single call to the FCP.

29. The data storage querying system of any preceding Claim 27 - 28 in which the data required to respond to a query is contiguously located in the file-store in a single key range so that it can be returned efficiently in a single call to the FCP.

30. The data storage querying system of any preceding Claim 27 - 29 in which after the optimal query plan has been determined, a single call to the data store via FCP is made withthe list of key prefixes or ranges identified in the optimal plan, and the FCP returns all matching results.

31. The data storage querying system of any preceding Claim 28 - 30 in which the FCP is always called with discrete key ranges and no wildcards.

32. The data storage querying system of any preceding Claim 28 - 31 in which there is a maximum of 1 FCP call per client request.Query Planner: Aggregation33. The data storage querying system of any preceding Claim in which the query planner subsystem is configured to aggregate matching results to reduce the number of results, before presenting the results to the user.

34. The data storage querying system of preceding Claim 33 in which, once matching results have been returned by FCP, the final stage is to perform group-by aggregation, grouping the results as defined by the request grouping option and ranking the results.

35. The data storage querying system of any preceding Claim 33 - 34 in which the query planner subsystem is configured to identify when a new file store is needed to avoid the need to aggregate matching quotes or results.

36. A data management system configured for generating results for user queries, the system including:(i) a query ingestion subsystem configured to receive end-user queries;(ii) a query planner subsystem configured to determine the most efficient way to generate a result to a request or query;(iii) a data store storing data that is queried by the query planner;(iv) a call management system that controls requests to external data sources;(v) an API that provides an interface to the external data sources;(vi) a data serving system that receives and processes queries from end-users and serves results to end-users;in which the data store is structured for retrieval of stored data, and the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys;and the query planner subsystem is configured to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold.

37. A method for improving the processing and querying of a data store, comprising: (a) structuring a data store structured for retrieval of stored data, in which the stored data is organised into multiple file stores, and in which each file store is described by a key made up of a number of key components, and each file store has a unique and different key that uses an ordered combination of the key components that is different from the ordered combination of key components used for other file stores, and the data store enables the fast retrieval of data given a key or ranges of keys or multiple ranges of keys;(b) operating a query planner subsystem to build key trees for all file stores grouped with the same rank, each tree representing a different query plan and to stop for any key tree that exceeds a width threshold.

Images