Secure data retrieval

Abstract

The present disclosure is directed towards an encrypted data structure configured to be stored on one or more servers including: a storage portion configured to store a set of data, wherein: the storage portion is configured to store data encrypted using a first algorithm; and an operations portion is configured to store the set of data, wherein: the operations portion is configured to store data encrypted using an operation encryption algorithm, and an operation encryption algorithm is an encryption algorithm that allows the performance of at least one predetermined operation on data in a data set that has been encrypted with the operation encryption algorithm, wherein the predetermined operation can be performed without decryption of the data.

Description

[0001] SECURE DATA RETRIEVAL

[0002] Field

[0003] The present application relates to secure data retrieval.

[0004] Background

[0005] In today's digital age, the security of data has become a paramount concern for organizations across various industries. With the increasing reliance on databases for storing and managing sensitive information, there is a growing need for robust and efficient encryption techniques to protect this data from unauthorized access and potential breaches. One such technique involves the use of encrypted queries for secure data retrieval from databases.

[0006] Structured query language (SQL) is a widely used programming language for managing and manipulating relational databases. SQL allows users to perform various operations on the data stored in databases, such as inserting, updating, deleting, and retrieving data. However, traditional SQL queries operate on plaintext data, which can be vulnerable to security threats and data breaches.

[0007] To address these concerns, various encryption techniques have been developed to protect sensitive data stored in databases. These techniques include column-level encryption, application layer encryption, and database-level encryption. While these methods provide a certain level of security, they often come with limitations, such as reduced functionality, increased complexity, and performance overheads.

[0008] One promising approach to overcome these limitations is the use of encrypted SQL queries for secure data retrieval. This method offers several advantages over traditional encryption techniques. For instance, it requires minimal re-engineering of existing software applications, as the query rewriting process is performed dynamically at runtime. This approach also offers superior security, as the encrypted data and metadata are protected from unauthorized access, even if the database or server is compromised. Furthermore, encrypted SQL queries can provide better performance by allowing the use of database indexes on encrypted data and enabling efficient search and compute operations on encrypted data.

[0009] Despite these advantages, there are still challenges and limitations associated with the use of encrypted SQL queries for secure data retrieval. For example, the implementation of searchable encryption and homomorphic encryption techniques are considered to be complex and their use at present requires specialized cryptographic expertise. Additionally, the integration of encrypted SQL queries with existing technology stacks and database systems may require significant effort and customization.

[0010] Improvements are desired to overcome these shortcomings and further enhance the security and efficiency of data retrieval using encrypted SQL queries.

[0011] Summary of the Invention

[0012] The present disclosure is directed towards an encrypted data structure configured to be stored on one or more servers including: a storage portion configured to store a set of data, wherein: the storage portion is configured to store data encrypted using a first algorithm; and an operations portion is configured to store the set of data, wherein: the operations portion is configured to store data encrypted using an operation encryption algorithm, an operation encryption algorithm is an encryption algorithm that allows the performance of at least one predetermined operation on data in a data set that has been encrypted with the operation encryption algorithm, wherein the predetermined operation can be performed without decryption of the data, and the first encryption algorithm and the operation encryption algorithm are different algorithms.

[0013] Preferably, the operation encryption algorithm performs at least one of homomorphic, deterministic, or order preserving encryption.

[0014] Preferably, the encrypted data structure is a table, the storage portion is a column, and the operations portion is a column.

[0015] Preferably, the encrypted data structure further comprises a second operations portion, configured to store the set of data, wherein: the second operations portion is configured to store data encrypted using a second operation encryption algorithm, wherein the second operation encryption algorithm is different from the first operation encryption algorithm.

[0016] Preferably, at least one operation encryption algorithm is selected for use based on a classification of the data that the encrypted data structure is configured to store.

[0017] The present disclosure is also directed towards a computer readable storage medium comprising a data structure according to any preceding claim. Further, the present disclosure is also directed towards a computer program including instructions which, when executed by a processor coupled to memory, cause the processor to generate a data structure as described above.

[0018] Preferably, the instructions are data management commands, such as, for example, SQL commands.

[0019] The present disclosure is also directed towards a configuration file comprising a specification for a data structure as described above, wherein the specification is used to determine one or more operation encryption algorithms for the data structure.

[0020] The present disclosure is also directed towards a method of retrieving encrypted data from an encrypted data structure, the method comprising: receiving a data management command at a database driver; converting the data management command into an encrypted data management command, wherein: converting the data management command into the encrypted data management command comprises configuring the encrypted data management command to operate on an operations portion associated with at least one storage portion of an encrypted data structure, wherein: the storage portion comprises a set of data containing at least one data entry encrypted with an encryption algorithm; and the operations portion includes the set of data, wherein the set of data is encrypted with an operations encryption algorithm, wherein the operations encryption algorithm is an encryption algorithm that allows the performance of a predetermined operation on data in a data set which has been encrypted by the operations encryption algorithm without decrypting the data.

[0021] Preferably, converting the data management command into an encrypted data management command comprises encrypting at least a portion of the data management command using an operations encryption algorithm.

[0022] Preferably, the encrypted data management command is sent from the database driver to a data storage system.

[0023] Preferably, converting the data management command comprises generating an encrypted data management command based on the data management command and the contents of a configuration file.

[0024] Preferably, converting the data management command comprises: using a syntax analysis engine to generate an abstract syntax tree data structure; locating a command in the abstract syntax tree data structure and amending at least a portion of the command based on the contents of a configuration file; generating an amended abstract syntax tree data structure wherein the command is replaced by the amended command; and generating an encrypted data management command based on the amended abstract syntax tree data structure.

[0025] Preferably, converting the amended abstract syntax tree data structure into an encrypted data management

[0026] Brief Description

[0027] The invention will be more clearly understood from the following description of an embodiment thereof, given by way of example only, with reference to the accompanying drawings, in which:-

[0028] Figure 1 shows a prior art data structure and a data structure in accordance with the present disclosure; and

[0029] Figures 2 and 3 show a method in accordance with the present disclosure.

[0030] Detailed Description

[0031] The present disclosure is directed towards enabling users of a system to utilise advanced encryption technologies (for example searchable encryption, homomorphic encryption, or these two techniques combined) without needing to know anything about these encryption techniques. I.e. The resultant system should ideally require “Zero Additional Learning” And “Zero Cryptographic Expertise.” For example, when configuring a system, administrators (such as database or application administrators) should merely have to decide whether a particular data type (e.g. a particular table or column) within a database should be encrypted (as well as specifying the users, user groups, or users and user groups that require access to the associated data type).

[0032] Ideally, such system integrates with existing technology to facilitate retrofitting of a system according to the present disclosure to existing applications and systems. Such systems need to be configurable to work with systems that comprise a backend relational database system that supports SQL (structure query language). While the solution provided by the present disclosure can be applied to any suitable data management and query language, for the sake of clarity the present disclosure is illustrated using the example of SQL because SQL is an industry standard that is ubiquitous in the field of databases. The solution provided by the present disclosure has been found to be particularly well suited to use with relational database systems. However, the use of the solution provided by the present disclosure is also envisaged for use with non-relational database systems. A system in accordance with the present disclosure is particularly suitable for use with an application that accesses a database using SQL commands transmitted using a database driver, such as e.g. a clientserver application. As a result, utilisation of a system in accordance with the present disclosure requires minimal re-engineering of existing software applications.

[0033] In particular, the present application is directed towards an encryption engine that is configured to dynamically rewrite a data management command (e.g an SQL or NoSQL command) from its original plaintext format to an encrypted format that can be used to query encrypted data. In the context of the present disclosure, a “Plaintext Format” data management command (PT- DMC) is a command (e.g. an SQL or NoSQL command) that operates on a set of database entries, wherein the entries do not contain an encrypted data type (e.g. a database where the tables or columns of the database are plaintext and are not encrypted at a table or column level). “Encrypted Format” data management commands (E-DMCs) are discussed below in more detail. However, it should be noted that an E-DMC can be easily and extensively configured by a system administrator.

[0034] Although there are existing methods for encrypting the columns or tables of a database, these existing solutions require the manual rewriting of all existing data management commands (DMCs) which reference a recently encrypted column or table. This needs to be done to adapt the DMCs for use with the encryption used within that column or table. The manual rewriting of the DMCs typically requires a user to have knowledge of encryption, as well as cryptographic key management. In addition, manually rewriting a DMC also increases the length of the DMC, which in turn increases the complexity of the resulting DMC.

[0035] In order to overcome this problem, a software library (which is also known as a library) is provided. A shared library or shared object is a file that is intended to be shared by executable files and further shared object files. Modules used by a program are loaded from individual shared objects into memory at load time or runtime, rather than being copied by a linker when it creates a single monolithic executable file for the program. The software library of the present disclosure is configured to rewriting PT-DMCs to produce E-DMCs. The library is preferably encapsulated within a modified database driver. Preferably a modified database drivers comprising the library is compatible with industry standard database connectivity APIs, e.g. Java Database Connectivity (JDBC). As a result of the driver being compatible with standard database connectivity APIs, a modified database drivers in accordance with the present disclosure can be used as a direct, like- for-like replacement for existing database drivers. A further advantage of generating E-DMCs from PT-DMCs in a modified database driver is that it is typically easy to replace a database driver - e.g. replacing a prior art database driver with a modified database driver according to the present disclosure will typically require a single line of source code being changed.

[0036] Normal APIs are preferred because they obviate the need for software engineers to reengineer database access code to use custom APIs. Instead, the shared library in the modified database driver obviates the need to rewrite DMC (e.g. SQL) source code for column-level or table-level encryption.

[0037] Preferably, a system according to the present disclosure comprises a database encryption management tool (DEMT). The DEMT is an application which is used to manage the encryption of a database. For example, the DEMT can be used to configure a database - e.g. to add or remove encrypted columns or tables from a database.

[0038] Preferably, the DEMT is configured to connect to a database and obtain information about the database model - i.e. the DEMT obtains data indicative of the logical structure of the database. For example, the DEMT may obtain a list of tables present in a database. Alternatively, or in addition, the DEMT may obtain a list of columns within one or more tables. The DEMT can also be used to create a database - in this case the data indicative of the logical structure of the database is obtained from a user.

[0039] The DEMT presents the organizational structure of data to be stored to a user. The user is then able to configure the structure by selecting a storage element (which also referred to herein as a storage portion) and configuring it to store either plaintext data or encrypted data. A storage element can be any organizational component in which data will be stored. For example, a user can select a database from a set of databases, a particular table in a database, or column in a particular table as a storage element and configure it to store either plaintext or encrypted data. It should be noted that if a database or a table is configured to store encrypted data, then the columns in the database of table will be configured to store encrypted data.

[0040] When the user has configured the organizational structure, the DEMT generates a Data Definition Language (DDL). The DDL comprises a subset of DMCs which, when executed, create a new storage element (such as, for example, a new database, a new database table, or a new database column). In particular, the DEMT generates a set of DMCs which may be used to generate the "encrypted" version of the new storage element. The set of DMCs can be sent directly to a target database server and executed immediately. Alternatively, the set of DMCs may be output as a executable script or executable file format to be executed later (e.g. using one or more DLLs).

[0041] With reference to figure 1, when a plaintext storage element 10 (such as e.g. a column in a database table) is encrypted, a set of encrypted storage elements 20 is created. The set of encrypted storage elements includes at least one operations storage element 21. The operations operations element 21 stores the data of the plaintext storage element in an encrypted form, where the data has been encrypted to allow at least one operation to be performed on the encrypted data in the operations storage element without its decryption.

[0042] For example, the operations storage element 21 may be an encrypted equality storage element (enc_eq). The enc_eq storage element is used to store data that has been encrypted in a manner that allows an equality check to be performed. I.e. the enc_eq enables the system to perform a determination of whether two encrypted values are identical and return a value indicative of the determination (e.g. true or false). The data stored in the enc_eq storage element is deterministically encrypted. Preferably, a hash-based message authentication code (HMAC) is used within the enc_eq storage element.

[0043] As a further example, the operations storage element 21 may be an encrypted comparison storage element (enc_comp). The enc_comp storage element is used to enable the comparison of two encrypted values - i.e. the enc_comp storage element contains data that has been encrypted in a manner that allows the system to compare two values (e.g Value A and Value B) and return a value indicative of the comparison (e.g. A<B, A=B, or A>B). Further, the enc_comp storage element allows data in the database table to be sorted in ascending or descending order. In addition, the enc_comp storage element also preferably facilitates comparing entries for equality. The data stored in the enc_comp storage element is encrypted using order preserving encryption. Preferably, an order preserving hash function is used.

[0044] As a further example, the operations storage element 21 may be an encrypted wildcard search storage element (enc_wild). The enc_wild storage element allows the system to compare an encrypted value with an encrypted wildcard search query. A wildcard search query is a value which comprises a wildcard. The wildcard indicates that at least one portion of the value is unknown. The comparison of an encrypted value and an encrypted wildcard search query is outlined in co-pending application EP23196506.2. In addition, enc_wild storage element also enables an equality check. The data stored in the enc_wild storage element is deterministically encrypted. Preferably, a hash-based message authentication code (HMAC) is used within the enc_wild storage element. Further, the enc_wild storage element is configured to store one or more encrypted values in each entry in the storage element. If the storage element is a column, each row may store one or more encrypted values.

[0045] In particular, an entry in the enc_wild storage element 21 preferably stores one or more modified permuterm index entries. A modified permuterm index entry is generated from a permuterm index entry, where the permuterm index entry comprises X values including a delimiting value inserted after the last value of the keyword. A modified permuterm index entry comprises the first Y values of the permuterm index entry appended with a zero-or-more wildcard value, Y is less than or equal to X.

[0046] As a further example, the operations storge element 21 may be an encrypted arithmetic storge element (enc_ar). The enc_ar storge element is encrypted in a manner that allows arithmetic operations to be performed on the data in the enc_ar storge element. Preferably, the data stored in the enc_ar storge element is homomorphically encrypted.

[0047] Preferably, one of more operations storge elements 21 are assigned to a plaintext storge element on encryption based on the type of data in the plaintext storge element. For example, an enc_ar storge element is preferably provided when the data in the plaintext storge element is numeric data - e.g: int, float, double, etc. In addition, an enc_comp storge element can also be provided to a plaintext storge element of numeric data. Thus, a plaintext storge element may, on encryption, be provided with i) an enc_comp storge element, ii) an enc_ar storge element, or ill) an enc_comp storge element and an enc_ar storge element.

[0048] Similarly, a plaintext storge element of temporal data (e.g. time and / or dates) may, on encryption, be provided with i) an enc_comp storge element, ii) an enc_ar storge element, or ill) an enc_comp storge element and an enc_ar storge element. A plaintext storge element of string data (i.e. text) can, on encryption, be provided with i) an enc_comp storge element, ii) an enc_wild storge element, or ill) an enc_comp storge element and an enc_wild storge element. Other plaintext storge elements may be provided with other suitable operations storge element(s) (e.g. and enc_eq storge element) based on the data stored in the plaintext storge element. Optionally, in addition to operations storge elements being assigned to a plaintext storge element based on the type of data in the plaintext storge element, a user (such as a system administrator) can add or remove operation storge element. This enables a user to cryptographically limit what operations can be performed on encrypted data. This provides the user with better control than merely limiting operations that can be performed by another user based on access controls. Further, this also can be used to reduce the size of the encrypted database by removing operation storage elements that are not required.

[0049] Preferably, a secondary encrypted storage element 22 stores the plaintext storage element data in an encrypted form. Any suitable encryption technique can be used. Preferably, any suitable symmetric encryption algorithm known in the art may be used.

[0050] Optionally, metadata (for example, names such as database names, table names, or column names) may be encrypted to improve security. However, metadata may be left unencrypted to enable backward compatibility with existing database systems.

[0051] The DEMT also creates an encryption configuration file (ECF). The ECF is a data structure that identifies the storage elements in a storage system that are encrypted. The ECF is provided to a modified database driver. The modified database driver uses the ECF to identify the storage elements of the storage system (e.g. a database, a table, a column, etc.) that are to be encrypted at run time. This in turn is used to identify that the data that needs to be encrypted when being sent to the storage system or decrypted on receipt. In this way the ECF is used to substantively encapsulate any changes to a database schema or table schema that have been made by a user using the DEMT.

[0052] While the DEMT is used to implement changes at the storage system level, the database driver uses the shared library and the ECF to identify these changes in runtime. In this way the database driver is able to rewrite a PT-DMC as an E-DMC. This rewriting is preferably done at the application level. In addition, this rewriting can be done dynamically.

[0053] In one configuration, an ECF is provided for the storage system. Alternatively, an ECF can be provided for one or more databases in the storage system. As a further alternative, it is also possible to provide an ECF for one or more tables in a database in the storage system.

[0054] The ECF preferably includes the plaintext name of the element of the storage system that has been specified as being encrypted by the DEMT. The ECF also identifies the type of data (e.g. numerical, temporal, text, etc) stored in the element. Any customisations applied to the element of the storage system (e.g. customised columns) is also stored in the ECF. The ECF also stores any configurations which have been applied to the element (e.g. whether or not the metadate is encrypted or not).

[0055] Turning to the stored library, in order to enable a storage system comprising at least one database to perform arithmetic operations on homomorphically encrypted numerical data, there are significant performance issues with stored functions or stored procedure extensibility mechanisms when they are written using a procedural DMC and stored within the storage system itself. To avoid these issues, a system in accordance with the present disclosure uses native functions to expand the functionality of the storage system. Native functions are preferably implemented and stored separate from the storage system through the use of a shared library such as a DLL or one or more shared objects etc. In this case, the user (e.g. a system administrator) enables a function to be invoked directly from the storage system using the DEMT and ECF.

[0056] When another user submits a plaintext query that requires the performance of a function on encrypted data (e.g. an arithmetic function on encrypted numerical data), the database driver which sends the query to the server. The server is configured to use the shared library to rewrite the query as an encrypted query. The encrypted query as rewritten is configured to invoke an encrypted function on the appropriate operations column for the encrypted data (e.g. the operations column associated with the column referred to in the original query - such as an enc_ar column for arithmetic operations). The encrypted function is configured to perform the function specified in the original query, but as the encrypted function is performed in the encrypted domain, it may not be the same function as that specified in the original query. To put it differently, if two values A and B are stored in encrypted form E(A) and E(B) and an encrypted function EF() is performed on two encrypted values - then the result of the operation E(C) when decrypted should match the result of performing the operation F specified in the query on A and B in plaintext - e.g. EF(E(A),E(B)) = E(C) where F(A,B)=C. If the operation is addition, it is noted encrypting two numbers, adding the two encrypted number, and then decrypting the result will not match the result produced by adding the two unencrypted numbers - to put it differently, F(E(A),E(B)) does not equal EF(E(A),E(B)), so if F() is addition, E(A)+E(B) will not produce a result that, when it is decrypted, will equal A+B. Instead the corresponding function in the encrypted domain, EF(), needs to be used. Given a PT-DMC, the database driver first constructs an abstract syntax tree (AST) for that command. The database driver then traverses the abstract syntax tree for the command and alters one or more of the keywords, clauses, expressions, or predicates in the command based on the contents of the ECF. The amended command is then converted into an E-DMC which will perform the encrypted operation on encrypted data which is equivalent to performing the received plaintext operation on plaintext data.

[0057] For the sake of clarity, it is noted that an E-DMC is a syntactically correct DMC - i.e. an E-DMC is designed to be executed by existing data storage systems. As such, when the PT-DMC is an SQL command, the E-DMC will also be an SQL command. As such, standard DMC command will still be used in an E-DMC according to the present disclosure. For example, in the case of an SQL, commands such as INSERT, UPDATE, DELETE, and SELECT commands will still be present in an E-DMC command. Using SQL as an example, if the PT-DMC is SELECT name then the E-DMT might be SELECT E(name) - i.e. the target of the query (name) is replaced with its encrypted equivalent.

[0058] The follow pseudocode sets out a number of DMCs, as well as what the process of translating a PT-DMC to an e-DMC. For the sake of illustration, plaintext table and column names below have been used. However, as noted above, table and column names may be encrypted.

[0059] Selecting entries with a specific value from a column using a PT-DMC = SELECT (where the SELECT command is used return data entries in a table having values matching specified criteria):

[0060] • PT-DMC: SELECT col FROM table WHERE col = val;

[0061] • E-DMC: SELECT col_enc FROM table WHERE col_enc_eq = enc_eq_val;

[0062] In the above example, the PT-DMC is query and val is the plain text value being queried - i.e. in the case of the PT-DMC, the database driver will return the entries in the table having a value that equals val in column col. Importantly, the database driver uses the same built in (e.q. SQL) operator for both the E-DMC and the PT-DMC.

[0063] Selecting entries with a specific range from a column using a PT-DMC = SELECT

[0064] • PT-DMC: SELECT col FROM table WHERE col > val;

[0065] • E-DMC: SELECT col_enc FROM table WHERE col_enc_comp > enc_comp_val;

[0066] In the above example, the PT-DMC is query and val is the plain text value being queried - i.e. in the case of the PT-DMC, the database driver will return the entries in the table having a value that is greater that val in column col. Importantly, the database driver uses the same built in (e.q. SQL) operator for both the E-DMC and the PT-DMC. Although the greater than operator (>) has been used, it is noted that the less than operator (<) could also be used.

[0067] Selecting entries with that match a wildcard search from a column using a PT-DMC = SELECT

[0068] • PT-DMC: SELECT col FROM table WHERE col LIKE 'val';

[0069] • E-DMC: SELECT col_enc FROM table WHERE col_enc_wildcard = 'enc_wild_val';

[0070] In the above example, the PT-DMC is query and val is the plain text value being queried - i.e. in the case of the PT-DMC, the database driver will return the entries in the table having a value that matches the wildcard query val in column col. In this case the E-DMC specifies the use of the built- in equality operator (i.e. the LIKE operation used in the PT-DMC is not used for the E-DMC).

[0071] Arithmetic Query (SELECT - Aggregate Function)

[0072] • PT-DMC: SELECT SUM(col) FROM table;

[0073] • E-DMC: SELECT FHE_AGGREGATE_SUM(enc_ar_col) FROM table;

[0074] In the above example, the PT-DMC is a query that is performed on a plain text column col in a given table. In this case the query returns the sum of the of the values in column col (e.g. the built- in SQL SUM function). In this case the corresponding E-DMC is obtained by translating the SUM function into a predetermined function which is referred to in this disclosure as an FHE_AGGREGATE_SUM function. The FHE_AGGREGATE_SUM function operates on a column of enc_ar data created from col data as described above to produce an encrypted value which, when decrypted, equals the result of the SUM function performed on unencrypted data.

[0075] Arithmetic Query (SELECT - Scalar Function)

[0076] • PT-DMC: SELECT col_1 + col_ 2;

[0077] • E-DMC: SELECT FHE_SCALAR_SUM(enc_ar_ col_1, enc_ar_col_2);

[0078] In the above example, the PT-DMC is a query that is performed on plain text columns col_1 and col_2. In particular, the query returns a column of values where each row in the column is the value of the corresponding row of col_2 added to the corresponding row of col_1. In this case, the E- DMC translates the *+’ function into a predetermined function referred to as a FHE_SCALAR_SUM function. The FHE_SCALAR_SUM function operates on a first column of enc_ar data (en_ar_col_1) created from col_1 data and a second column of of enc_ar data (en_ar_col_2) created from col_2 data. INSERT Query

[0079] • PT-DMC: INSERT INTO table (col) VALUES (val);

[0080] • E-DMC: INSERT INTO table (col_enc, col_enc_eq, col_enc_comp, col_enc_wild, col_enc_ar) VALUES (enc_val, enc_eq_val, enc_comp_val, enc_wild_val, enc_ar_val);

[0081] In the above example, the PT-DMC is a command to insert a plain text value (val) into a column (col) of plain text values in a table. In this case the PT command is translated into an encrypted command to insert the one or more different types of encrypted values corresponding to the plain text value into one or more encrypted columns, where each encrypted column is for a different type of encrypted data to the other encrypted columns. As described above, the exact number of "additional” encrypted columns may vary depending on different factors - including the exact data type, as well as user preferences.

[0082] UPDATE Query

[0083] • Plaintext Query: UPDATE table SET col = val WHERE pk = 1;

[0084] • Encrypted Query: UPDATE table SET col_enc = enc_val, col_enc_eq = enc_eq_val, col_enc_comp = enc_comp_val, col_enc_wild = enc_wild_val, col_enc_ar = enc_ar_val WHERE pk = 1;

[0085] In the above example, the PT-DMC is a command to update a plain text value (val) in one or more rows of a first column (col) of plain text values in a table where the rows updated correspond to the rows in a second column (pk) of plain text values having a value of 1. In this case the PT command is translated into an encrypted command to update one or more different types of encrypted values corresponding to the plain text values being updated in one or more encrypted columns, where each encrypted column contains a different type of encrypted data from the other encrypted columns. As described above, the exact number of "additional columns" may vary depending on different factors - including the exact data type, as well as user preferences.

[0086] Figure 2 illustratively shows a system for performing a query according to the present disclosure, and Figure 3 illustratively shows a flowchart for performing a method 100 according to the present disclosure. The method 100 preferably includes six steps. The first step 110 comprises receiving a database query 101. The second step 120 comprises converting the query 101 using a syntax analysis engine 102 to generate an abstract syntax tree data structure 103. In the third step 130, a command 104 is located in the abstract syntax tree data structure 103 and at least a portion of the command 104 is amended based on the contents of a configuration file 105. In the fourth step 140, an amended abstract syntax tree data structure 107 is generated wherein the command 104 is replaced by the amended command. In the fifth step 150, the amended abstract syntax tree data structure is converted into an encrypted database query 108. In the sixth step 160, the encrypted query 108 is used to retrieve encrypted data.

[0087] Advantageously, the method 100 enables organizations to leverage cutting-edge encryption technologies, such as searchable encryption and homomorphic encryption, without requiring any prior knowledge or expertise in cryptography. This is accomplished by seamlessly integrating with existing technology stacks, which allows the solution to be retrofitted into current applications and systems with minimal disruption. For instance, the method 100 can be easily integrated into legacy systems, cloud-based applications, or even mobile applications, providing a wide range of compatibility and adaptability.

[0088] The solution is specifically designed to work with software systems that consist of a backend relational database system that supports SQL (Structured Query Language) and a software application that accesses the database using DMCs transmitted via a database driver, such as a client-server application. This compatibility ensures that the method 100 can be employed in various industries and sectors, including finance, healthcare, government, and e-commerce, where secure data storage and retrieval are of paramount importance.

[0089] In more detail, the method 100 allows organizations to utilize searchable encryption, which enables efficient searching of encrypted data without the need for decryption. This feature is particularly beneficial for large-scale databases, where decrypting the entire dataset for a single query would be computationally expensive and time-consuming. For example, a healthcare organization could use searchable encryption to securely search for patient records based on specific criteria, such as a medical condition or a particular treatment, without exposing sensitive patient information.

[0090] Additionally, the method 100 incorporates homomorphic encryption, which allows mathematical operations to be performed directly on encrypted data without the need for decryption. This capability is advantageous for organizations that require secure data processing and analysis, such as financial institutions that need to perform calculations on encrypted financial data for risk assessment or fraud detection purposes. By utilizing homomorphic encryption, these organizations can maintain the confidentiality of their data while still being able to perform necessary computations. The integration of the method 100 with existing technology stacks is achieved through the use of a database driver that is compatible with standard database connectivity apis, such as Java JDBC. This compatibility allows the suite of database drivers to be used as direct, like-for-like replacements for existing database drivers, ensuring a smooth transition to the new encryption solution. In most cases, this process involves changing only a single line of source code, making it easy for organizations to adopt the method 100 without extensive re-engineering efforts.

[0091] Furthermore, the method 100 provides a high level of customization and flexibility, as it can be tailored to meet the specific security requirements of an organization. For example, the method 100 can be configured to encrypt only certain columns or tables within a database, allowing organizations to protect their most sensitive data while maintaining the performance of less critical operations. This selective encryption capability ensures that organizations can strike the right balance between security and performance, depending on their unique needs and priorities.

[0092] In the method 100, minimal re-engineering of existing software applications is required. For example, a predetermined library, can be encapsulated within a database driver. The predetermined enables the database driver to dynamically rewrite DMCs from their original plaintext format to an encrypted format. By implementing standard database connectivity APIs, such as Java JDBC, database drivers according to the present disclosure can be used as direct, like- for-like replacements for existing database drivers. This process typically involves changing only a single line of source code, making it easy to adopt the solution without extensive re-engineering efforts.

[0093] The method 100 also offers superior security by automatically encrypting data and metadata as it exits a database driver and decrypting it as it enters the database driver after being retrieved from the backend database. This automated encryption and decryption process takes place transparently within the database driver, ensuring that the encrypted database or the server hosting it never has access to the associated cryptographic key material. The cryptographic key material can be persisted using a tpm (trusted platform module), hsm (hardware security module), or a centralized kms (key management system) or secret management engine, such as hashicorp vault.

[0094] Furthermore, the method 100 offers superior performance by avoiding the need to decrypt all records in an encrypted table or column in order to search that table or column. This results in significant performance gains compared to traditional application layer / level encryption solutions and database / server-side column-level encryption solutions. The solution also allows for database indexes to be used on encrypted data. Database indexes are often lost once data is encrypted, resulting in significant performance loss. The use of operations column allows a unique combination of searchable encryption and homomorphic encryption to be used which in turn enables common operations such as searching values, checking values for equality, or comparing / ordering values to be performed efficiently.

[0095] The method 100 consists of four components: a database driver, a shared library, a key storage mechanism, and a database encryption management tool (DEMT). The DEMT is used for initial configuration and ongoing maintenance, enabling vaultree's technology to be used with a database and allowing administrators to add or remove encrypted tables and columns from a vaultree-enabled database. The DEMT produces two outputs: a set of DLL DMCs (such as SQL commands) for creating a new database table and an encryption configuration file, which is used by the database driver to identify what databases, tables, and columns need to be encrypted or decrypted at runtime.

[0096] In the encrypted schema generated by the DEMT, additional columns are added to the encrypted version of a database table schema, depending on the data type of the associated column and the settings configured by the system administrator. These additional columns enable specific operations to be performed on encrypted data, such as encrypted equality, encrypted comparison, encrypted wildcard search, and encrypted arithmetic. The solution also supports the encryption of metadata, such as database names, table names, and column names.

[0097] In summary, the method 100 provides a comprehensive solution for organizations to implement advanced encryption technologies in their existing systems with minimal reengineering efforts, superior security, and performance.

[0098] It will be understood that the term "encrypted data" as used herein may refer to information that has been transformed into a coded format using encryption algorithms, rendering it unreadable without the appropriate decryption key, such as the data stored in the database mentioned in the invention.

[0099] It will be understood that the term “E-DMC” as used herein may refer to a modified DMC, such as a Structured Query Language (SQL) command, that has been transformed and encrypted to securely access and retrieve data from an encrypted database, in accordance with the disclosed invention. It will be understood that the term "amended command" as used herein may refer to a modified version of the original command in the abstract syntax tree, which has been altered based on the configuration file to enable the retrieval of encrypted data from the database.

[0100] It will be understood that the term "amended abstract syntax tree data structure" as used herein may refer to a modified representation of the hierarchical structure of a DMC (such as an SQL query), where the original abstract syntax tree has been altered based on a configuration file to facilitate the retrieval of encrypted data from a database.

[0101] It will be understood that the term "retrieve encrypted data" as used herein may refer to the process of accessing and obtaining information from a database that has been secured and stored in an encrypted format, typically through the use of cryptographic algorithms and techniques.

[0102] It will be appreciated by the person of skill in the art that various modifications may be made to the above-described examples without departing from the scope of the invention as defined by the appended claims.

Claims

Claims1. An encrypted data structure configured to be stored on one or more servers including: a storage portion configured to store a set of data, wherein: the storage portion is configured to store data encrypted using a first encryption algorithm; and an operations portion is configured to store the set of data, wherein: the operations portion is configured to store data encrypted using an operation encryption algorithm, an operation encryption algorithm is an encryption algorithm that is selected to allows the performance of at least one predetermined operation on data in a data set that has been encrypted with the operation encryption algorithm, wherein the predetermined operation can be performed without decryption of the data, and the first encryption algorithm and the operation encryption algorithm are different algorithms.

2. The encrypted data structure of claim 1, wherein the operation encryption algorithm performs at least one of homomorphic, deterministic, or order preserving encryption, or performs an encryption algorithm that allows an encrypted wild card search to be performed.

3. The encrypted data structure of claim 1 or 2, wherein the encrypted data structure is a table, the storge portion is a column, and the operations portion is a column.

4. The encrypted data structure of any one of claims 1 to 3, further comprising a second operations column, configured to store the set of data, wherein: the second operations portion is configured to store data encrypted using a second operation encryption algorithm, wherein the second operation encryption algorithm is different from the first operation encryption algorithm.

5. The encrypted data structure of any one of claims 1 to 4, wherein at least one operation encryption algorithm is selected for use based on a classification of the data that the encrypted data structure is configured to store.

6. A computer readable storage medium comprising a data structure according to any preceding claim.

7. A computer program including instructions which, when executed by a processor coupled to memory, cause the processor to generate a data structure according to any one of claims 1 to 5.

8. The computer program of claim 7, wherein the instructions are data management commands.

9. The computer program of claim 8, wherein the data management commands are SQL commands.

10. A configuration file comprising a specification of for a data structure defined by any one of claims 1 to 5, wherein the specification is used to determine one or more operation encryption algorithms for the data structure.

11. A method of retrieving encrypted data from an encrypted data structure, the method comprising: receiving a data management command at a database driver; converting the data management command into an encrypted data management command, wherein: converting the data management command into the encrypted data management command comprises configuring the encrypted data management command to operate on an operations portion associated with at least one storage portion of an encrypted data structure, wherein:the storage portion comprises a set of data containing at least one data entry encrypted with an encryption algorithm; and the operations portion includes the set of data, wherein the set of data is encrypted with an operations encryption algorithm, wherein the operations encryption algorithm is an encryption algorithm that allows the performance of a predetermined operation on data in a data set which has been encrypted by the operations encryption algorithm without decrypting the data.

12. The method of claim 11, wherein converting the data management command into an encrypted data management command comprises encrypting at least a portion of the data management command using an operations encryption algorithm.

13. The method of claim 11 or 12, comprising sending the encrypted data management command from the database driver to a data storage system.

14. The method any one of claims 11 to 13, wherein converting the data management command comprises generating an encrypted data management command based on the data management command and the contents of a configuration file.

15. The method of any one of claims 11 to 14, wherein converting the data management command comprises: using a syntax analysis engine to generate an abstract syntax tree data structure; locating a command in the abstract syntax tree data structure and amending at least a portion of the command based on the contents of a configuration file; generating an amended abstract syntax tree data structure wherein the command is replaced by the amended command; and generating an encrypted data management command based on the amended abstract syntax tree data structure.

16. The method of any one of claims 9 to 14, wherein converting the amended abstract syntax tree data structure into an encrypted data management command.

Images