A database indexing method supporting field-level key online rotation and related device

Abstract

The application relates to a database index method supporting field-level key online rotation and related equipment. The method comprises the following steps: in response to an instruction of converting a first key version of a target field into a second key version, setting a key rotation state as a migration state; in the migration state, a target field of newly written data is encrypted by using the second key version, and a target field encrypted by using the first key version is gradually migrated to be written back after being encrypted by using the second key version; in the migration state, if an equivalent query request for the target field is received, a first query token and a second query token are respectively generated; a first result set is obtained by searching in a first index structure by using the first query token, and a second result set is obtained by searching in a second index structure by using the second query token; the result sets are combined and deduplicated, and a final query result is fed back. The method improves data query accuracy, improves key rotation update efficiency, and guarantees business continuity of a database.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a database indexing method and related equipment that supports online rotation of field-level keys. Background Technology

[0002] Document databases have been widely used in the internet, finance, and government sectors due to their flexible support for semi-structured data such as nested documents and array fields. With the improvement of data security regulations and the increasing awareness of personal privacy protection, implementing fine-grained encryption at the field level for sensitive fields in document databases (such as ID card numbers, bank card numbers, and mobile phone numbers), and maintaining efficient equivalent query capabilities in the encrypted state, has become a key technical problem that urgently needs to be solved in the industry.

[0003] Database indexing schemes supporting field encryption in related technologies mainly fall into two categories. One approach involves encrypting the field at the application layer before writing it to the database, where only the encrypted data is stored. In this scheme, the database cannot directly use the encrypted field to create an index, leading to a significant performance degradation in equality queries targeting the encrypted field. Furthermore, when the key used for the encrypted field needs to be updated, coordination among application systems is difficult, resulting in complex and costly key rotation operations. The other approach integrates field-level encryption and query rewriting capabilities into the database kernel, enabling unified management of encryption strategies. However, when the key for the encrypted field needs to be updated, this scheme typically requires pausing database writes or performing system-wide maintenance, and offline reconstruction of all indexes related to the encrypted field, severely impacting business continuity.

[0004] There are currently no effective solutions to the problems of poor data query accuracy, low key rotation and update efficiency, and the impact of key rotation and update on database business continuity in related technologies. Summary of the Invention

[0005] The present invention provides a database indexing method and related equipment that supports online rotation of field-level keys, which at least partially solves the problems of poor data query accuracy, low key rotation update efficiency, and the impact of key rotation updates on the business continuity of the database in related technologies.

[0006] To address the aforementioned problems, one aspect of this invention provides a database indexing method that supports online rotation of field-level keys, comprising: In response to a key rotation instruction for a target field in a document database, the key rotation status corresponding to the target field is set to a migration state; wherein, the key rotation instruction is used to instruct the conversion of the first key version corresponding to the target field to a second key version; in the migration state, the target field in newly written data is encrypted using the second key version, and the target field in the document database encrypted using the first key version is gradually migrated and updated to be re-encrypted using the second key version and then written back for storage; In the migration state, if an equivalence query request for the target field is received, the query value in the equivalence query request is obtained, and a first query token is generated based on the first key version and the query value, and a second query token is generated based on the second key version and the query value; wherein, the retrieval in the first index structure and the retrieval in the second index structure are executed in parallel; The first query token is used to search in the first index structure associated with the first key version to obtain a first result set that matches the first query token, and the second query token is used to search in the second index structure associated with the second key version to obtain a second result set that matches the second query token; The first result set and the second result set are merged and deduplicated to generate and return the final query result.

[0007] In some embodiments, the method further includes: The original field path of the target field is standardized to map the dynamic path containing array subscripts to a standardized field path that ignores the differences in array subscripts; An index structure associated with the key version is established based on the standardized field path; wherein, the first index structure contains index entries associated with the first key version, the second index structure contains index entries associated with the second key version, and the index entries are the mapping relationship between the query token corresponding to the target field and the document identifier, and the document identifier is used to uniquely identify the document data containing the target field.

[0008] In some embodiments, the method further includes: During the migration process, an asynchronously executed data migration task scans and reads historical document data belonging to the target field, which is encrypted using the first key version, from the historical data stored in the document database. For each historical document containing the target field, the following operations are performed: decrypt the target field in the historical document using the first key version to obtain the plaintext to be migrated; re-encrypt the plaintext to be migrated using the second key version to obtain the ciphertext to be migrated; generate a query token to be migrated based on the second key version and the plaintext to be migrated; write back and store the ciphertext to be migrated to the ciphertext storage location corresponding to the historical document in the document database; store the query token to be migrated in the second index structure to update the index entry of the historical document in the second index structure; and delete the index entry corresponding to the historical document in the first index structure.

[0009] In some embodiments, before the steps of writing back and storing the ciphertext to be migrated to the ciphertext storage location corresponding to the historical document data in the document database, and storing the query token to be migrated in the second index structure, the method further includes: Detect the version indication information corresponding to the historical document data; wherein, the version indication information is used to characterize the document version status of the historical document data in the document database; If the version indication information is detected to have changed compared to the version indication information before the data migration task was executed and scanned, it indicates that the historical document data has been updated during the data migration process, and the write-back storage operation is abandoned. The latest version of the historical document data is rescanned and read to obtain the target field ciphertext and its encryption key version corresponding to the latest version data. If the encryption key version is the second key version, the write-back storage operation of the historical document data is skipped. If the encryption key version is the first key version, the corresponding query token to be migrated and the ciphertext to be migrated are re-determined based on the plaintext value obtained after decrypting the target field ciphertext.

[0010] In some embodiments, the document database includes master nodes and slave nodes, and the method further includes: When the master node generates the data update log, it writes the current key version information, grace key version information, current key rotation status information, ciphertext data migration update result, and query token update result corresponding to the target field into the data update log. The slave node replays the data update log and reconstructs local data based on the current key version information, the grace key version information, the current key rotation status information, the ciphertext data migration update result, and the query token update result, so as to ensure that the data status between the slave node and the master node is consistent.

[0011] In some embodiments, the method further includes: Monitor the migration progress of historical document data. In response to determining that all target fields encrypted with the first key version have been migrated and updated to be re-encrypted with the second key version and written back to storage, and that the slave node has been synchronized to the same data state as the master node, update the key rotation status corresponding to the target field to the migration completed status, and deactivate the first key version.

[0012] In some embodiments, the method further includes: Before receiving the key rotation instruction, the plaintext value of the target field is stored in the document data of the document database in ciphertext form, and the corresponding query token is stored in an index structure separate from the document data, so as to decouple ciphertext storage from index retrieval.

[0013] In some embodiments, the first index structure and the second index structure are equality index structures that support equality matching retrieval based on query tokens, and the equality index structures include hash index structures, inverted index structures, and key-value index structures.

[0014] To address the aforementioned problems, one aspect of this invention provides a database indexing device that supports online rotation of field-level keys, comprising: The status setting module is used to respond to a key rotation instruction for a target field in the document database and set the key rotation status corresponding to the target field to a migration state; wherein, the key rotation instruction is used to instruct the conversion of the first key version corresponding to the target field to a second key version; in the migration state, the target field in newly written data is encrypted using the second key version, and the target field in the document database encrypted using the first key version is gradually migrated and updated to be re-encrypted using the second key version and then written back for storage; A query token generation module is used to, in the migration state, if an equivalence query request for the target field is received, obtain the query value in the equivalence query request, and generate a first query token based on the first key version and the query value, and generate a second query token based on the second key version and the query value; wherein, the retrieval in the first index structure and the retrieval in the second index structure are executed in parallel; The parallel query module is used to search in a first index structure associated with the first key version using the first query token to obtain a first result set matching the first query token, and to search in a second index structure associated with the second key version using the second query token to obtain a second result set matching the second query token. The result feedback module is used to merge and deduplicate the first result set and the second result set, and generate and feedback the final query result.

[0015] To address the aforementioned problems, one aspect of this invention provides a non-transitory machine-readable medium storing computer instructions for causing a computer to execute any of the aforementioned database indexing methods that support online rotation of field-level keys.

[0016] The beneficial effects of this invention are as follows: In response to a key rotation instruction for a target field in a document database, the key rotation status corresponding to the target field is set to a migration state. The key rotation instruction is used to instruct the conversion of the first key version corresponding to the target field to a second key version. In the migration state, the target field in newly written data is encrypted using the second key version, and the target field in the document database encrypted with the first key version is gradually migrated and updated to be re-encrypted with the second key version before being written back for storage. In the migration state, if an equivalence query request for the target field is received, the query value in the equivalence query request is obtained, and a first query token is generated based on the first key version and the query value. A second query token is generated based on the second key version and the query value. The first query token is used to perform a search in the first index structure associated with the first key version to obtain a first result set matching the first query token. The second query token is used to perform a search in the second index structure associated with the second key version to obtain a second result set matching the second query token. The search in the first index structure and the search in the second index structure are executed in parallel. The first and second result sets are merged and deduplicated to generate and return the final query result. This invention overcomes the problems of poor data query accuracy, low key rotation update efficiency, and the impact of key rotation updates on database business continuity in related technologies. By responding to key rotation commands and setting the key rotation status to a migration state, it uses dual-version query tokens to retrieve two sets of index structures in parallel and merges and deduplicates the results. This achieves the goal of quickly and accurately obtaining equivalent query results without downtime or waiting for all historical data migration to complete, thus improving query efficiency, data query accuracy, key rotation update efficiency, and ensuring database business continuity.

[0017] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the main process of a database indexing method that supports online rotation of field-level keys, according to one embodiment of the present invention. Figure 2 This is a schematic diagram of the main flow of a database indexing method in a migration state according to one embodiment of the present invention; Figure 3 This is a schematic diagram of the main modules of a database indexing device that supports online rotation of field-level keys, according to one embodiment of the present invention. Figure 4 This is a schematic diagram of partial module interactions of a database indexing device according to one embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the invention. Detailed Implementation

[0020] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0021] To more clearly illustrate the solutions of the embodiments of the present invention, the main technical terms involved in the embodiments of the present invention are briefly explained as follows: Query token: A query token is a string calculated using a deterministic token generation algorithm based on the plaintext value of the target field and a specified key version. It is used for equality matching retrieval within the index structure. Specifically, the same plaintext value yields the same query token under the same key version, while different key versions yield different query tokens. For example, the plaintext card number "12"... 56” calculates token A under the first key version and token B under the second key version; the two are not the same.

[0022] Plaintext value: The actual data content of the target field stored in the document data, such as the bank card number "12". 56" or ID number "20260101××××". In this embodiment of the invention, the plaintext value is the object that the user expects to match when querying, but it needs to be encrypted and protected at the storage level.

[0023] Ciphertext: Data obtained by encrypting a plaintext value using an encryption algorithm with a specified key version. In this embodiment of the invention, the ciphertext is the storage object corresponding to the target field. The ciphertext cannot be read directly and must be decrypted using the correct key version to restore the plaintext value. For example, the bank card number "12" 56” is encrypted using the first key version to obtain the binary string “0x7A3F…”.

[0024] Index structure: This refers to a data structure that stores the mapping relationship between query tokens and document identifiers, supporting the rapid location of matching document identifiers based on query tokens. Specifically, the first index structure corresponds to the first key version, and the index entries in the first index structure store query tokens generated based on the first key version; the second index structure corresponds to the second key version, and the index entries in the second index structure store query tokens generated based on the second key version.

[0025] Original field path: The access path from the root node to the target field in the document data. For example, in a nested document {"user":{"cards":[{"number":"12 56"},{"number":"78 In 12", the original field path corresponding to the first card number is user.cards.0.number, and the original field path corresponding to the second card number is user.cards.1.number. Different array indices result in the same semantic field corresponding to different original field paths.

[0026] Standardized field path: The unified path representation obtained after normalizing the original field path, ignoring the differences in array subscripts, uniformly maps user.cards.0.number and user.cards.1.number to user.cards[].number.

[0027] Document Identifier: A field in a document database used to uniquely identify a document, such as the "_id" field in a MongoDB database. Every document has a unique document identifier.

[0028] To address the aforementioned problems, embodiments of the present invention provide a database indexing method that supports online rotation of field-level keys, such as... Figure 1 As shown, this database indexing method that supports online rotation of field-level keys mainly includes: Step S101: In response to the key rotation instruction for the target field in the document database, the key rotation status corresponding to the target field is set to the migration state; wherein, the key rotation instruction is used to indicate that the first key version corresponding to the target field is converted to the second key version; in the migration state, the target field in the newly written data is encrypted with the second key version, and the target field in the document database encrypted with the first key version is gradually migrated and updated to be re-encrypted with the second key version and then written back for storage; Step S102: In the migration state, if an equivalence query request for the target field is received, the query value in the equivalence query request is obtained, and a first query token is generated based on the first key version and the query value, and a second query token is generated based on the second key version and the query value. Step S103: Using the first query token, a search is performed in the first index structure associated with the first key version to obtain a first result set matching the first query token; and using the second query token, a search is performed in the second index structure associated with the second key version to obtain a second result set matching the second query token; wherein, the search in the first index structure and the search in the second index structure are performed in parallel. Step S104: Merge and deduplicate the first result set and the second result set to generate and return the final query result.

[0029] Based on the above settings, during the online rotation of field-level keys in the document database, when encrypted data corresponding to the old and new key versions coexist, the two sets of index structures are retrieved in parallel by using dual-version query tokens, and the results are merged and deduplicated. This allows for the rapid and accurate acquisition of equivalent query results without downtime or waiting for the migration of all historical data to be completed. This achieves the technical effects of improving query efficiency, enhancing data query accuracy, increasing key rotation and update efficiency, and ensuring business continuity of the database.

[0030] The first key version and the second key version mentioned above are not specific versions; they are only relative. When a key rotation instruction is received, the first key version refers to the current key version used to encrypt the target field in the document database. After receiving the key rotation instruction, the first key version represents the key version that needs to be phased out (the old version key), and the second key version represents the key version that needs to be adopted immediately (i.e., the new key version).

[0031] The aforementioned target fields refer to sensitive fields (such as ID card numbers, bank card numbers, mobile phone numbers, and other fields involving users' personal privacy information), or fields that require encrypted querying based on actual application needs. Different target fields can independently maintain their key rotation status, and different key fields can be in different key rotation stages.

[0032] Specifically, based on step S101 above, upon receiving the key rotation instruction, the key rotation status corresponding to the specified target field is set to "migration in progress." This allows the target field in newly written data to be encrypted using the second key version (i.e., the new key), while the target fields in the document database already encrypted with the first key version (i.e., the old key) enter a gradual migration process. In the "migration in progress" state, the database kernel knows that the target field's current key rotation status is in a transition period between the old and new key versions. It decomposes the key rotation process into two parallel operation paths: "immediately switching the target field in newly written data to the new key version" and "gradually migrating the target fields in historical data encrypted with the old key version to the new key version for re-encryption and writing back to storage." This allows the key rotation to start without waiting for all historical data migration to complete, avoiding downtime. Simultaneously, the "migration in progress" status flag enables the database to distinguish between normal and transitional states, thereby executing different query routing strategies in different states, ensuring the accuracy and comprehensiveness of data queries.

[0033] According to a specific embodiment of the present invention, the essence of the above-mentioned migration state is to allow the document database to accept two key versions simultaneously in the time dimension for a specific field (i.e., the target field indicated by the key rotation instruction). For example, assuming there is a document set D in the database, where the subset of documents whose target field is encrypted using the first key version is D_V1, and the subset of documents whose target field is encrypted using the second key version is D_V2, then in the migration state, D = D_V1 ∪ D_V2. In the migration state, the target field in newly written data is directly encrypted using the second key version and directly stored in D_V2. The historical documents stored in the original D_V1 whose target field is still encrypted using the first key version are gradually updated to use the second key version through a background migration task. After the migration is completed, the document is transferred from D_V1 to D_V2.

[0034] According to another specific embodiment of the present invention, a path strategy table can be constructed in the document database to include path strategy items such as the current key version, grace key version, and key rotation status. When a query operation is performed, the path strategy items can be read first to determine the indexing strategy. If the key rotation status in the path strategy item is in a migration state, a dual-version index routing strategy can be triggered (i.e., searching in both the first and second index structures simultaneously).

[0035] In some embodiments, based on the migration state described in step S102 above, for the received equivalent query request, two different query tokens are generated based on the query value and the two key versions respectively. This realizes the transformation of a single query value into query tokens under two different key versions, enabling subsequent retrieval to simultaneously cover index entries encrypted with the first key version and index entries encrypted with the second key version, providing query-level preparation for the complete retrieval of document data in both the old and new versions.

[0036] According to a specific embodiment of the present invention, the query token is a deterministic function, denoted as Token=F(plaintext value, key version). For the same plaintext value v, when the first key version K1 is input, the first query token Token1=F(v,K1) is obtained; when the second key version K2 is input, the second query token Token2=F(v,K2) is obtained. Since K1 and K2 are different, and the token generation algorithm satisfies collision resistance (i.e., different input combinations have a very high probability of producing different outputs, and the same input combination produces the same output), Token1 and Token2 are different. Based on the above step S102, based on the same query value, two index retrieval key values ​​(i.e., two query tokens, Token1 and Token2) are generated respectively through the first key version and the second key version, so that subsequent queries can hit the corresponding index entries in the two index structures respectively, ensuring comprehensive coverage of the query range. Among them, since the query token generation process is completed entirely in the database kernel, the user does not need to be aware of the current migration state. Therefore, the user only submits an equality query request for the target field and its query value, and the database kernel automatically executes the dual query token generation according to the current state.

[0037] In some examples, based on step S103 above, parallel retrieval using a dual-index structure is used to obtain the matching document set encrypted with the first key version and the matching document set encrypted with the second key version, respectively, avoiding data omissions caused by retrieving only a single index structure during migration.

[0038] The first index structure and the second index structure are physically or logically independent index structures, each maintaining its own index entries. The index entries in the first index structure store the mapping relationship between query tokens and document identifiers generated based on the first key version. That is, the set of query tokens in the first index structure is T_V1={F(value,K1)|value∈target field value in document data}; the set of query tokens in the second index structure is T_V2={F(value,K2)|value∈target field value in document data}.

[0039] It should also be noted that while parallel retrieval using a dual-index structure can significantly improve query efficiency and accuracy, there are still some extreme cases that can lead to incomplete query coverage and missed results. Specifically, when two query tasks are submitted simultaneously, scenarios where query vulnerabilities may occur due to factors such as thread scheduling, index response time, and different submission times of data migration tasks. For example, if other factors cause a query based on the second index structure to be executed first, followed by a query based on the first index structure, when the query based on the second index structure is read first, the data migration task has not yet written the index entries of the second index structure, and the retrieval of the second index structure does not obtain the corresponding document results. The data migration task then writes the index entries of the second index structure corresponding to the target field after updating the encryption key version and deletes the corresponding index entries in the first index structure (i.e., the data migration task performs migration for the target field). When the query based on the first index structure is read again, since the index entries in the first index structure have been deleted, the retrieval of the first index structure also does not obtain the corresponding document results, resulting in an extreme vulnerability where neither index returns the document query results corresponding to the target field.

[0040] To overcome potential omissions and help ensure that query omissions are reduced or avoided, according to a specific embodiment of the present invention, the parallel retrieval steps of the above-mentioned dual-index structure may further include performing dual-version index retrieval based on the same consistent reading snapshot / reading timestamp, or by first establishing a second version index entry, overlapping and retaining the first version index entry, and deleting the first version index entry after the query barrier is satisfied, so that queries during data migration can hit at least one valid index entry before or after data migration; then the results returned by the two indexes are merged and deduplicated according to the document identifier, thereby reducing or avoiding query omissions and duplicate returns during the migration switching process.

[0041] According to another specific embodiment of the present invention, the step of performing dual-version index retrieval based on the same consistent read snapshot / read timestamp may include: obtaining a consistent read snapshot or read timestamp, and under the consistent read snapshot or one-time read timestamp, using a first query token to perform a retrieval in a first index structure associated with a first key version to obtain a first result set matching the first query token, and using a second query token to perform a retrieval in a second index structure associated with a second key version to obtain a second result set matching the second query token.

[0042] In some cases, based on step S104 above, the two result sets are merged and deduplicated, eliminating any duplicate documents that may exist in the two result sets, and returning a complete and non-redundant final query result, so that the query result obtained by the user is semantically consistent with the query result before the key rotation.

[0043] During migration, some document data may have entries in both index structures due to index update delays, but the document identifier of the same document is the same in different index structures (the uniqueness of the document identifier). By merging and deduplicating, the document identifiers from the two index structures are merged, and duplicate document identifiers are eliminated.

[0044] In some embodiments, the method further includes: standardizing the original field path of the target field to map the dynamic path containing array subscripts to a standardized field path that ignores array subscript differences; and an index structure based on the standardized field path and associated with a key version; wherein the first index structure contains index entries associated with a first key version, the second index structure contains index entries associated with a second key version, the index entries are the mapping relationship between the query token corresponding to the target field and the document identifier, and the document identifier is used to uniquely identify the document data containing the target field.

[0045] Based on the above settings, path standardization unifies the semantically identical but path-different dynamic fields in the document database into standardized paths that can be recognized by the policy. Based on these standardized paths, index structures are built for the two key versions respectively. This allows newly written data and migration write-back data in the migration state during subsequent key rotation to be directly written into the corresponding index structures. This enables query requests to directly use the dual index structure for retrieval without the need for path identification, index creation, or index reconstruction during the rotation period.

[0046] The establishment of the first index structure generally occurs before the key rotation instruction, to ensure that the query result can be determined based on the first query token before the key rotation is performed. The establishment or activation of the second index structure is performed after the second key version is generated. According to a specific implementation of the present invention, the second index structure can be established / activated separately after the second key is generated (that is, the second index structure and the first index structure are independent physical indexes), or a second index partition can be added to the same physical index (that is, the first index structure and the second index structure belong to different partitions under the same physical index; after the second key is generated, a new partition can be created directly under the same physical index).

[0047] Specifically, according to embodiments of the present invention, by uniformly mapping a large number of semantically identical but syntactically different dynamic field paths generated by different array indices in the document database into a stable standardized field path, the target field with the same semantics can be expressed by a unique path identifier, avoiding strategy matching failures or omissions caused by path diversity.

[0048] The path standardization process described above establishes a mapping function from the original field path space to the standardized field path space. For any original field path of the same semantic field (e.g., user.cards.0.number, user.cards.1.number, user.cards.2.number), this mapping function outputs the same standardized field path (user.cards[].number). This path standardization process identifies and removes array subscripts, a hierarchical information that contributes nothing to semantics, while preserving the hierarchical relationship of field names. Specifically, array fields (e.g., cards) in document databases are stored using array subscripts to distinguish different elements. However, these subscripts are positional identifiers, not semantic identifiers (e.g., subscript 0 and subscript 1 only indicate different positions of the elements in the array, not different semantic fields). Therefore, in paths containing array subscripts, the differences in the subscript portion are redundant information for encryption strategy matching. The standardization process eliminates this redundant information by replacing array subscripts with wildcard representations (e.g., []), allowing the same field in different positions to have a unified expression at the path level. Accordingly, the index structure built based on standardized field paths can cover all storage instances of the target field at any array position, while also improving the efficiency of index structure building.

[0049] In some embodiments, before the key rotation begins, two index structures are pre-created based on standardized field paths, one associated with the first key version and the other with the second key version (the creation of the second index structure must also satisfy the requirement that the second key version has been generated). This provides a structural carrier for write and retrieval operations involving dual-version indexes during the rotation process, eliminating the need to temporarily create indexes or rebuild indexes offline during the rotation period.

[0050] In some examples, by limiting the index entries to a mapping between the query token and the document identifier corresponding to the target field, the document identifier is used to uniquely identify the document data containing the target field, thus clearly defining the specific content stored in the index entries (the mapping pair between the query token and the document identifier), enabling retrieval without reading and decrypting the data itself.

[0051] It's also understandable that index entries store the mapping between query tokens and document identifiers, making the retrieval operation essentially a key-value matching operation. Key-value matching is an efficient operation supported by the index structure (time complexity O(log n) or O(1)), far superior to the linear scanning method of decrypting document data one by one and then comparing it (time complexity O(n)). Furthermore, since index entries only store query tokens and not plaintext values, even if the index structure itself is accessed without permission, attackers cannot deduce the plaintext values, ensuring the security of the target field at the index level.

[0052] In some of these embodiments, the first index structure and the second index structure described above are equality index structures that support equality matching retrieval based on query tokens. The equality index structures include hash index structures, inverted index structures, and key-value index structures.

[0053] Based on the above settings, an example of an index structure type is provided. By adopting an index structure type that supports equality matching retrieval based on query tokens, the index structure can complete equality matching retrieval of the target field without reading or decrypting the ciphertext of the field stored in the document, thereby achieving efficient ciphertext index retrieval while ensuring data confidentiality.

[0054] The inverted index structure establishes a mapping from key values ​​to a list of document identifiers centered around the key value. The encrypted inverted index structure inherits this indexing method, replacing the plaintext key value with a query token, but the mapping between key values ​​and document identifiers remains unchanged. Since the query token itself is a deterministic byte sequence (e.g., SM4 encryption outputs fixed-length binary data), its physical storage form as a key value is essentially no different from the storage form of the plaintext key value at the data structure level (the index structure does not care whether the key value is encrypted; it only treats the key value as a comparable byte sequence for storage and indexing). Therefore, using the encrypted inverted index structure can leverage the key value location mechanism of the inverted index to complete equality matching retrieval based on ciphertext values ​​in constant time complexity.

[0055] According to embodiments of the present invention, the uniqueness of the document identifier ensures that the corresponding document data can be accurately located from the index entry. The index structure stores the document identifier, not the document data itself. This separation of index and data architecture allows the index structure to store only smaller key values ​​and identifiers, while storing larger document data elsewhere, thereby improving the index's retrieval efficiency. Specifically, the document identifier exists as a value in the index entry. When a key matches, the index structure returns this value, and the database then uses this value to locate and read the complete document data.

[0056] It is understood that, since the retrieval operation of the index structure used in this embodiment of the invention does not depend on the decryption of the stored ciphertext value (the ciphertext is responsible for storage, and the query token is responsible for retrieval), the entire query matching process is completed in the ciphertext state, thereby avoiding the computational overhead and data exposure risk caused by the decryption operation.

[0057] In some embodiments, the method further includes: during the migration process, by asynchronously executing a data migration task, scanning and reading historical document data belonging to the target field encrypted with a first key version from the historical data stored in the document database; for each historical document data containing the target field, performing the following operations: decrypting the target field in the historical document data using the first key version to obtain plaintext to be migrated, re-encrypting the plaintext to be migrated using a second key version to obtain ciphertext to be migrated, generating a query token to be migrated based on the second key version and the plaintext to be migrated; writing back the ciphertext to be migrated to the ciphertext storage location corresponding to the historical document data in the document database, and storing the query token to be migrated in the second index structure to update the index entry of the historical document data in the second index structure; and deleting the index entry corresponding to the historical document data in the first index structure.

[0058] Based on the above settings, a specific implementation method for data migration in the migration state is provided. In the migration state, through asynchronously executed migration tasks, the target fields in the document database encrypted with the first key version are gradually migrated to the second key version, so that the key version of the target field is smoothly transitioned, the migration process does not block the front-end business, and the migrated data is complete and consistent at both the document data and index levels.

[0059] According to a specific embodiment of the present invention, each migration operation of historical document data containing a target field includes three interrelated update contents, which are completed through the same atomic transaction or equivalent atomic commit mechanism. The first path is the data layer update path: read the old ciphertext → decrypt to obtain plaintext → re-encrypt with the new key to obtain new ciphertext → write back to the ciphertext storage location of the document data; the second path is the index layer addition path: generate a new query token based on the new key and plaintext → write to the second index structure → create an index entry under the new version; the third path is the index layer deletion path: delete the old index entry of the document in the first index structure. The data layer update path ensures that the key version of the encrypted target field is changed; the index layer addition path ensures that the document data to which the target field belongs after the key version update used for encryption can be retrieved by queries based on the second key version; the index layer deletion path ensures that the updated document data will not be repeatedly retrieved by queries based on the first key version.

[0060] According to another specific embodiment of the present invention, when it is determined that the rotation state corresponding to the target field is in the migration state, that is, when it is determined that a dual-index structure retrieval needs to be performed, the consistent read snapshot identifier or read timestamp corresponding to this retrieval is obtained. The first index retrieval corresponding to the first query token and the second index retrieval corresponding to the second query token are both executed based on the same consistent read snapshot identifier or read timestamp. During the same atomic transaction or equivalent atomic commit, the background migration task writes the ciphertext of the field corresponding to the second key version, establishes the index entry corresponding to the second key version, and invalidates or deletes the index entry corresponding to the first key version. Thus, based on the same read snapshot identifier or read timestamp, what is observed is the complete first version state before the migration commit or the complete second version state after the migration commit, and an intermediate state where neither version index entry exists is observed. The above-mentioned "atomic commit" can be implemented by a database transaction when the document data and index structure are in the same transaction domain, or by a commit identifier, version visibility control, or other atomic switching mechanism with equivalent effect provided by the database kernel.

[0061] In some embodiments, limiting the data migration task to asynchronous execution steps means that the data migration task and the front-end business operations (writing, querying, updating) run in different execution threads or processes. This decouples the migration operation of historical document data from the front-end business operations, allowing the database to still respond to write and query requests normally during the migration process and avoiding the migration operation blocking the business.

[0062] In some examples, after completing the data migration of the current historical data document, removing the old index entries remaining in the first index structure of the historical document data can effectively prevent the historical document data from being retrieved repeatedly in subsequent queries, maintain the uniqueness and correctness of the query results, and improve retrieval efficiency.

[0063] In some embodiments, before writing back the ciphertext to be migrated to the ciphertext storage location corresponding to the historical document data in the document database and storing the query token to be migrated in the second index structure, the method further includes: detecting version indication information corresponding to the historical document data; wherein, the version indication information is used to characterize the document version status of the historical document data in the document database; if it is detected that the version indication information has changed compared to the version indication information before the data migration task was scanned and read, it indicates that the historical document data has been updated during the data migration process, and the write-back storage operation is abandoned; rescanning and reading the latest version data of the historical document data, obtaining the target field ciphertext and its encryption key version corresponding to the latest version data, if the encryption key version is the second key version, the write-back storage operation of the historical document data is skipped; if the encryption key version is the first key version, the corresponding query token to be migrated and the ciphertext to be migrated are re-determined based on the plaintext value obtained after decrypting the target field ciphertext.

[0064] Based on the above settings, by comparing and detecting the version indication information, it can be identified whether the document to be written back by the data migration task has been updated in the front end. If it has been updated, the write-back will be abandoned and the migration will be re-executed based on the latest version of the data, thereby avoiding the back end migration data from overwriting the latest data written in the front end.

[0065] Specifically, there is a time window between the data migration operation and the write-back process. The length of this time window mainly depends on the computation time of the decryption and re-encryption operations. Within this time window, the front-end application may initiate an update request for the document and complete the write-back. If the migration task writes back directly without checking, the problem of lost updated documents will occur (i.e., the data updated by the front-end application is overwritten by the old data held by the migration task, and the front-end application is unaware of this overwriting, resulting in data inconsistency).

[0066] According to a specific embodiment of the present invention, a version indication information detection step is inserted before the write-back operation is executed. By comparing whether the current version indication information is consistent with the version indication information before scanning, it is accurately determined whether the document has been updated during the migration process (since the version indication information is modified with each update and only changes when the document is updated, there is no missed detection or false detection). When a document version update is detected, the migration of the document is not abandoned, but the migration operation is re-executed based on the latest data, thereby ensuring that the document can eventually complete the migration from the first key version to the second key version. It is understood that even if a document triggers version conflicts multiple times due to frequent updates, the migration task will continue to retry until the write-back is completed based on the latest data, and ultimately all historical document data will be migrated to the second key version.

[0067] It should also be noted that, based on the above settings, the embodiments of the present invention also enable the background data migration task and the foreground business operation to be executed concurrently securely without a lock mechanism. That is, the data migration task does not need to lock the document to prevent the foreground from writing, and the foreground writing does not need to wait for the data migration task to complete. The two can be executed independently, and the access to the same document is coordinated only by checking the version indication information when writing back, thereby ensuring data consistency without sacrificing concurrency performance.

[0068] The version indication information mentioned above is used to characterize the version status of document data in the document database, such as document version number, update timestamp, or logical sequence number. Version indication information has a monotonically increasing property; each time a document is updated, its version indication information changes, and the new value is greater than the old value. For example, the document version number changes from 5 to 6, or the update timestamp changes from time T1 to time T2 (T2 is later than T1).

[0069] In some embodiments, the document database includes a master node and slave nodes, and the method further includes: when the master node generates a data update log, writing the current key version information, grace key version information, current key rotation status information, ciphertext data migration update result, and query token update result corresponding to the target field into the data update log; the slave node reconstructs local data based on the current key version information, grace key version information, current key rotation status information, ciphertext data migration update result, and query token update result by replaying the data update log, so as to make the data state between the slave node and the master node consistent.

[0070] Based on the above settings, in a master-slave replication environment, the key version semantic information of the master node is transmitted to the slave node through the data update log. This enables the slave node to accurately know the key version and rotation status corresponding to each write operation of the master node, thereby avoiding key version mismatch caused by the slave node inferring on its own and ensuring that the data status between the master and slave nodes remains consistent during the key rotation process.

[0071] The aforementioned data update log refers to the log recorded by the master node for each data change operation. Slave nodes synchronize data changes with the master node by reading and replaying this data update log. The data update log typically includes the operation type, operation object, and data content before and after the change. The aforementioned current key version information refers to the identifier of the key version used by the master node to encrypt the target field when performing a write operation, indicating which key version was used for encryption in the write operation corresponding to this log entry. The aforementioned grace period key version information refers to the identifier of the old key versions that are allowed to be simultaneously compatible when querying during migration, indicating the range of key versions allowed for querying when the slave node is in a migration state at this log position. The aforementioned current key rotation status information identifies the key rotation status (e.g., normal or migration state) of the document database when the master node performs this write operation. The aforementioned encrypted data migration update result refers to the encryption key update result corresponding to the target data in the data migration task. The aforementioned replay refers to the process where the slave node reads entries from the data update log and performs the same operation (e.g., insert, update, delete) locally to ensure that the slave node's data state is consistent with the master node.

[0072] Specifically, according to an embodiment of the present invention, the master node writes the ciphertext field update result, the query token update result, and the corresponding version metadata such as keyVersion (current key version information), graceKeyVersion (grace key version information), migrationEpoch (rotation batch number), and migrationState (current key rotation status information) into the data update log. When the slave node replays the data update log, it reconstructs the local data state according to the ciphertext update result, index token update result, and version metadata carried in the log.

[0073] According to a specific embodiment of the present invention, in the above-described master-slave replication mechanism, the current key version information, grace key version information, current key rotation status information, ciphertext data migration update result, and query token update result are written as structured fields of the data update log (in another specific embodiment, rotation batch, document version, operation sequence number, etc., can also be written into the data update log), and are written into the replication link together with the ciphertext update result and query token update result of the target field. That is, the version semantic information does not only exist in the master node's memory or path policy table, but is persisted and transmitted to the slave node as an inherent component of the data update log. When the slave node replays the data update log, it directly reads the version semantic information from the data update log and reconstructs the local data state accordingly, without relying on the local policy table to infer or guess the key version used by the master node when writing. Since the version semantic information is already embedded in the log, even if the master node completes key rotation, updates the policy table in memory, or even disables the old key version after the log is generated, the slave node can still obtain the complete version context at the time of writing when replaying the log, thus ensuring the consistency of data state between the master and slave nodes during key rotation.

[0074] In some embodiments, the method further includes: monitoring the migration progress of historical document data; in response to determining that all target fields encrypted with the first key version have been migrated and updated to be re-encrypted with the second key version and written back to storage, and that the slave nodes have been synchronized to the same data state as the master node, updating the key rotation status corresponding to the target field to the migration completed status, and disabling the first key version.

[0075] Based on the above settings, by jointly verifying the completion of migration of all target fields of the master node and the completion of data synchronization of the slave node, it is ensured that when the first key version is deactivated, neither the master nor the slave node depends on the old key version, thereby avoiding the problem of data unavailability or inconsistency between master and slave data caused by prematurely deactivating the old key version.

[0076] According to a specific embodiment of the present invention, the data on the slave node is a copy of the data on the master node. After the master node migration is completed, the slave node still holds a copy of the old data before the migration for a period of time. If the first key version is deactivated before the slave node completes synchronization, the slave node will be unable to complete subsequent log replay. That is, if the first key version is deactivated too early, it will cut off the slave node's ability to replay migration logs, causing the slave node to remain permanently in an inconsistent state. Therefore, based on the above embodiments of the present invention, it is ensured that as long as any node (whether master or slave) still stores valid data encrypted with the first key version, the first key version cannot be deactivated.

[0077] In some embodiments, the method further includes: before receiving the key rotation instruction, storing the plaintext value of the target field in ciphertext form in the document data of the document database, and storing the corresponding query token in an index structure separate from the document data, so as to decouple ciphertext storage from index retrieval.

[0078] Based on the above settings, before the key rotation begins, an architecture is established to store document data and index structure separately. This allows update operations on document data and index structure to be executed independently and without blocking each other during the subsequent key rotation process. This provides a clear architectural foundation for the two index structures to maintain index entries independently in dual-version index routing.

[0079] According to embodiments of the present invention, the plaintext value of the target field is converted into ciphertext and stored in the document data, so that even if the storage medium is accessed without authorization, an attacker cannot directly read the actual content of the target field. Furthermore, the information used for retrieval (query token) is stored separately from the document data itself, allowing retrieval operations to be completed directly based on the index structure without accessing the document data.

[0080] Meanwhile, since the goal of document data storage is to securely preserve the encrypted form of plaintext values, while the goal of index retrieval is to quickly locate document identifiers that match query conditions, their requirements for data format, storage method, and update frequency differ. Ciphertext storage focuses on the selection of encryption algorithms and the management of key versions, while index retrieval focuses on the organization of the index structure and retrieval efficiency. By decoupling ciphertext storage from index retrieval, updates to document data (such as decryption and re-encryption operations during key rotation) do not require rebuilding the entire index structure; only the index entry corresponding to the document needs to be updated. Similarly, optimizing or rebuilding the index structure does not require re-encrypting the ciphertext storage in the document data. This allows both to be updated and maintained independently, without mutual constraints.

[0081] like Figure 2 As shown, this embodiment of the invention also provides a specific implementation method for responding to user query requests in a migration state.

[0082] In this specific implementation, it is assumed that a financial institution's document database stores a large amount of user bank card information, where the target field is the user's bank card number, and the corresponding field path is user.cards.number. Since this target field stores sensitive data, the database has implemented field-level encryption for it and established a corresponding index structure. Initially, the target field is encrypted using a first key version (hereinafter referred to as V1), and the first index structure stores the mapping relationship between query tokens generated based on V1 and document identifiers. During operation, in response to security policy requirements, the encryption key for the target field needs to be rotated from version V1 to a second key version (hereinafter referred to as V2). The administrator sends a key rotation instruction to the database for this target field, and the database responds by setting the key rotation status to "migration in progress." In the "migration in progress" state, the database simultaneously executes two operation processes: on the one hand, the target field in all newly written document data is encrypted using V2, and the corresponding query token is written to the second index structure; on the other hand, the background asynchronous migration task scans the historical document data to which the target field, encrypted using V1, belongs in batches, decrypts the target field, re-encrypts it using V2, and updates the corresponding index entries. During this period, the document database contains both target fields encrypted with V1 and target fields encrypted with V2 in newly written data.

[0083] The specific implementation methods for responding to user query requests during migration mainly include: Step 1: During the migration process, the database receives the following equality query request from the user: Query all bank card numbers where cardNo equals " The document data is "02020202".

[0084] Step Two: The database first standardizes the target field path in the query request. The field path specified in the original query statement is user.cards.number. For the actual data stored in the document database, the same bank card number field may correspond to different original field paths in different documents, such as user.cards.0.number, user.cards.1.number, etc. The path standardization module maps the above dynamic paths containing array subscripts to the standardized field path user.cards[].number, which ignores the differences in array subscripts.

[0085] Step 3: The database reads the path policy corresponding to the standardized field path. The path policy table records the current key version of the target field: active=V2, grace key version: grace=V1, and key rotation status: state=MIGRATING (in progress). Based on this path policy, the database determines that the key query status corresponding to the target field is currently in a migration state, and a dual-version index routing method is needed to process this query.

[0086] Step 4: Retrieve the query value from the user's query statement in the database. The query value "02020202" is then tokenized based on both the grace key version V1 and the current key version V2. Specifically, the database will... "02020202" and V1 are input into the token generation algorithm to calculate the first query token (denoted as V1 token); at the same time, the query value " The token generation algorithm uses "02020202" and V2 together to calculate the second query token (denoted as V2token). Since V1 and V2 are different key versions, the values ​​of T_V1 and T_V2 are different.

[0087] Step 5: Perform dual-version index retrieval based on the same consistent read snapshot: The database obtains a consistent read snapshot and, under the consistent read snapshot, performs a retrieval in the first index structure associated with V1 using the first query token V1. The index entries stored in the first index structure are the mapping relationship between historical query tokens generated based on V1 and document identifiers. The database searches for index entries matching the V1 token in the first index structure to obtain the first result set (V1 document). This first result set (V1 document) contains all bank card numbers equal to "" that were stored before the migration and have not yet been migrated. The document identifier is "02020202". Simultaneously, the database uses the second query token (V2 token) to perform a retrieval in the second index structure associated with V2. The second index structure stores index entries that represent the mapping between query tokens generated based on V2 and document identifiers. The database searches the second index structure for index entries that match the V2 token, obtaining a second result set (V2 documents). This second result set (V2 documents) contains all newly written entries after the migration began, where the bank card number equals "02020202". The document identifier is "02020202". The two retrieval operations mentioned above can be executed in parallel without interference. Both the first and second index structures use encrypted inverted index structures, supporting equality matching based on query tokens. The retrieval process does not require reading the corresponding document data or performing any decryption operations.

[0088] Step Six: The database performs a merge and deduplication operation on the first and second result sets. Since some entries may exist in both index structures simultaneously during the migration process, the merge and deduplication operation combines the document identifiers from the two result sets to eliminate any duplicate document identifiers. After merging and deduplication, the database obtains complete, duplicate-free final query results and returns them to the user.

[0089] In the above process, the entire query processing flow is completed automatically in the database kernel. Field path standardization ensures that the same semantic field containing different array subscripts can be uniformly identified and processed. Path strategy reading ensures that the database can select an appropriate query routing strategy based on the current rotation status. Dual-version token generation ensures that the query can cover data from both the old and new key versions. Parallel retrieval using dual index structures improves query efficiency and accuracy. Merging and deduplication ensures that the final result is complete and free of redundancy. Users only need to initiate an equality query request, without needing to know that the database is currently undergoing key rotation, nor do they need to care about how many key versions or index structures exist at the underlying level, to obtain complete and accurate query results.

[0090] The database indexing method supporting online field-level key rotation provided in this embodiment of the invention, by responding to a key rotation instruction for a target field in the document database, sets the key rotation status corresponding to the target field to a migration state; wherein, the key rotation instruction is used to instruct the conversion of the first key version corresponding to the target field to a second key version; in the migration state, the target field in newly written data is encrypted using the second key version, and the target field in the document database encrypted with the first key version is gradually migrated, updated, re-encrypted with the second key version, and then written back for storage; in the migration state, if an equality query request for the target field is received, The process involves retrieving the query value from the equivalent query request, generating a first query token based on the first key version and the query value, and generating a second query token based on the second key version and the query value. The first query token is then used to search within a first index structure associated with the first key version to obtain a first result set matching the first query token. Similarly, the second query token is used to search within a second index structure associated with the second key version to obtain a second result set matching the second query token. The searches in the first and second index structures are performed in parallel. The first and second result sets are then merged and deduplicated to generate and return the final query result. This approach improves query efficiency, enhances data query accuracy, increases key rotation and update efficiency, and ensures business continuity of the database.

[0091] Based on the database indexing method supporting online rotation of field-level keys provided in the embodiments of the present invention, the embodiments of the present invention also provide a database indexing device supporting online rotation of field-level keys, such as... Figure 3 As shown, the database indexing device 300 that supports online rotation of field-level keys includes: The status setting module 301 is used to respond to the key rotation instruction for the target field in the document database and set the key rotation status corresponding to the target field to the migration state. The key rotation instruction is used to indicate that the first key version corresponding to the target field is converted to the second key version. In the migration state, the target field in the newly written data is encrypted with the second key version, and the target field in the document database that is encrypted with the first key version is gradually migrated and updated to be re-encrypted with the second key version and then written back to the storage. The query token generation module 302 is used to, in the migration state, if an equivalence query request for a target field is received, obtain the query value in the equivalence query request, and generate a first query token based on the first key version and the query value, and generate a second query token based on the second key version and the query value; wherein, the retrieval in the first index structure and the retrieval in the second index structure are executed in parallel; The parallel query module 303 is used to search in the first index structure associated with the first key version using the first query token to obtain a first result set matching the first query token, and to search in the second index structure associated with the second key version using the second query token to obtain a second result set matching the second query token. The result feedback module 304 is used to merge and deduplicate the first result set and the second result set, and generate and feedback the final query result.

[0092] Based on the above settings, the status setting module responds to the key rotation command by setting the key rotation status to "migration in progress"; the query token generation module determines the dual-version query token; the parallel query module searches the two sets of index structures in parallel based on the dual-version query token to obtain the results; and the result feedback module merges and deduplicates the results and provides the final query result. This achieves the goal of quickly and accurately obtaining equivalent query results without downtime or waiting for all historical data to be migrated, thereby improving query efficiency, data query accuracy, key rotation update efficiency, and ensuring business continuity of the database.

[0093] Meanwhile, since this application uses the database indexing device 300 that supports online rotation of field-level keys to implement any of the above-mentioned database indexing methods that support online rotation of field-level keys, the execution operations corresponding to the individual execution of each module in the database indexing device that supports online rotation of field-level keys, or the interaction between each module, are consistent with those in the aforementioned database indexing method that supports online rotation of field-level keys, and can achieve all the beneficial effects corresponding to the aforementioned database indexing method that supports online rotation of field-level keys. This application will not elaborate further here.

[0094] It should also be noted that the specific module settings within the database indexing device that supports online rotation of field-level keys are defined primarily based on the corresponding operations performed, and are not intended to limit the specific modules. For example, a key management module can be introduced into the database indexing device to store multiple versions of keys.

[0095] According to a specific implementation method of the present invention, such as Figure 4 As shown, the database indexing device is deployed in the document database kernel. The interaction process of the database indexing device during key rotation mainly includes: The administrator sends a key rotation command to the database indexing device through the management interface, which instructs the encryption key corresponding to the target field to be rotated from the first key version to the second key version.

[0096] After receiving the key rotation instruction, the master node requests a second key version (V2 key) from the key management module. The key management module responds to this request by generating or obtaining the second key version (V2 key) and returning it to the master node, ensuring that the second key version (V2 key) is different from and independent of the first key version (V1 key). The key management module is the functional component in the database indexing device responsible for key version lifecycle management. It maintains metadata information for each key version (including key version identifier, generation time, status, etc.) and provides services for key version application, distribution, and deactivation.

[0097] After the master node obtains the second key version, it sets the key rotation status corresponding to the target field to "Migration in Progress". In the "Migration in Progress" state, the status management module controls the database to perform two-way operations: The first operation is write path control. For the target field in the newly written data, the target field is encrypted using the second key version (V2 key), and the generated query token is written into the second index structure.

[0098] The second operation is migration path control, which triggers the background migration module to start the historical data migration task. After starting, the background migration module begins scanning and reading historical document data (V1 documents) containing the target field encrypted with the first key version from the historical data stored in the document database. For each historical document containing the target field: first, the ciphertext data of the target field in the historical document data is read; the first key version (V1 key) is requested from the key management module; and the ciphertext data is decrypted using the first key version (V1 key) to obtain the plaintext to be migrated. Subsequently, the background migration module requests the second key version (V2 key) from the key management module; the plaintext to be migrated is re-encrypted using the second key version (V2 key) to obtain the ciphertext to be migrated; and a query token to be migrated is generated based on the second key version and the plaintext to be migrated. The ciphertext to be migrated is used to update the ciphertext storage in the document data, and the query token to be migrated is used to update the index entries in the index structure. The background migration module writes back the ciphertext to be migrated to the corresponding ciphertext storage location in the document database and writes back the query token to be migrated to the second index structure to update the index entries of the historical document data in the second index structure. Simultaneously, the background migration module deletes the corresponding index entry for this historical document data in the first index structure. This migration operation continues until all target fields encrypted with the first key version are migrated, updated to use the second key version, and written back to storage.

[0099] Throughout the entire duration of the migration state, the query processing module maintains its ability to respond to user query requests. The interaction process of the query processing module is as follows.

[0100] When a user submits an equality query request for a target field to the database indexing device via the client, the query request is sent to the query processing module. The equality query request contains the user-specified query value, such as a bank card number. 02020202".

[0101] The query processing module corresponding to the master node can receive equivalent query requests. It standardizes the original field paths of the target field in the query request, mapping dynamic paths containing array indices (e.g., user.cards.0.number, user.cards.1.number) to standardized field paths that ignore array index differences (e.g., user.cards[].number). Subsequently, the query processing module reads the path policy corresponding to this standardized field path, obtains the current key rotation status information, and confirms that it is currently in a migration state. In the migration state, the query processing module uses a dual-version index routing method to process this query.

[0102] The query processing module corresponding to the master node obtains the query value from the user's query statement and requests the first key version and the second key version from the key management module. The query processing module generates a first query token (V1 Token) based on the first key version and the query value, and generates a second query token (V2 Token) based on the second key version and the query value.

[0103] Then, using the index retrieval module corresponding to the master node, a search is performed in the first index structure associated with the first key version based on the first query token (V1 Token) to obtain a first result set matching the first query token. At the same time, the index retrieval module performs a search in the second index structure associated with the second key version based on the second query token (V2 Token) to obtain a second result set matching the second query token.

[0104] Finally, the result merging module corresponding to the master node merges and removes duplicates from the first and second result sets, combines the document identifiers in the two result sets to eliminate duplicate document identifiers, generates the final query result, and returns the final query result to the query user.

[0105] It is understood that, based on the specific implementation methods described above, the embodiments of the present invention can obtain complete and accurate equivalent query results without downtime or waiting for all historical data migration to be completed during key rotation.

[0106] This invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this invention.

[0107] This invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the methods of embodiments of this invention. The computer program product should be understood as a software product that primarily implements the methods of this invention through a computer program.

[0108] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of this invention.

[0109] refer to Figure 5 The present invention will now be described in the form of a structural block diagram of an electronic device that can serve as an embodiment of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0110] like Figure 5 As shown, the electronic device includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0111] Multiple components in the electronic device are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information into the electronic device. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0112] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0113] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of embodiments of the present invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] It should be noted that the term "comprising" and its variations used in the embodiments of the present invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that, unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0116] The information / data involved in the embodiments of this invention (including but not limited to user device information / data, user personal information / data, information / data used for analysis, stored information / data, displayed information / data, etc.) are all information / data that has been permitted by the user or fully agreed upon by all parties. Furthermore, the collection, use and processing of such information / data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to agree or refuse.

[0117] The steps described in the method embodiments provided by this invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of this invention is not limited in this respect.

[0118] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0119] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A database indexing method supporting online rotation of field-level keys, characterized in that, include: In response to a key rotation instruction for a target field in a document database, the key rotation status corresponding to the target field is set to a migration state; wherein, the key rotation instruction is used to instruct the conversion of the first key version corresponding to the target field to a second key version; in the migration state, the target field in newly written data is encrypted using the second key version, and the target field in the document database encrypted using the first key version is gradually migrated and updated to be re-encrypted using the second key version and then written back for storage; In the migration state, if an equivalence query request for the target field is received, the query value in the equivalence query request is obtained, and a first query token is generated based on the first key version and the query value, and a second query token is generated based on the second key version and the query value. The first query token is used to search in a first index structure associated with the first key version to obtain a first result set matching the first query token, and the second query token is used to search in a second index structure associated with the second key version to obtain a second result set matching the second query token; wherein the search in the first index structure and the search in the second index structure are performed in parallel. The first result set and the second result set are merged and deduplicated to generate and return the final query result.

2. The method according to claim 1, characterized in that, The method further includes: The original field path of the target field is standardized to map the dynamic path containing array subscripts to a standardized field path that ignores the differences in array subscripts; An index structure associated with the key version is established based on the standardized field path; wherein, the first index structure contains index entries associated with the first key version, the second index structure contains index entries associated with the second key version, and the index entries are the mapping relationship between the query token corresponding to the target field and the document identifier, and the document identifier is used to uniquely identify the document data containing the target field.

3. The method according to claim 1, characterized in that, The method further includes: During the migration process, an asynchronously executed data migration task scans and reads historical document data belonging to the target field, which is encrypted using the first key version, from the historical data stored in the document database. For each historical document containing the target field, the following operations are performed: decrypt the target field in the historical document using the first key version to obtain the plaintext to be migrated; re-encrypt the plaintext to be migrated using the second key version to obtain the ciphertext to be migrated; generate a query token to be migrated based on the second key version and the plaintext to be migrated; write back and store the ciphertext to be migrated to the ciphertext storage location corresponding to the historical document in the document database; store the query token to be migrated in the second index structure to update the index entry of the historical document in the second index structure; and delete the index entry corresponding to the historical document in the first index structure.

4. The method according to claim 3, characterized in that, Before the steps of writing back and storing the ciphertext to be migrated to the ciphertext storage location corresponding to the historical document data in the document database, and storing the query token to be migrated in the second index structure, the method further includes: Detect the version indication information corresponding to the historical document data; wherein, the version indication information is used to characterize the document version status of the historical document data in the document database; If the version indication information is detected to have changed compared to the version indication information before the data migration task was executed and scanned, it indicates that the historical document data has been updated during the data migration process, and the write-back storage operation is abandoned. The latest version of the historical document data is rescanned and read to obtain the target field ciphertext and its encryption key version corresponding to the latest version data. If the encryption key version is the second key version, the write-back storage operation of the historical document data is skipped. If the encryption key version is the first key version, the corresponding query token to be migrated and the ciphertext to be migrated are re-determined based on the plaintext value obtained after decrypting the target field ciphertext.

5. The method according to claim 1, characterized in that, The document database includes master nodes and slave nodes, and the method further includes: When the master node generates the data update log, it writes the current key version information, grace key version information, current key rotation status information, ciphertext data migration update result, and query token update result corresponding to the target field into the data update log. The slave node replays the data update log and reconstructs local data based on the current key version information, the grace key version information, the current key rotation status information, the ciphertext data migration update result, and the query token update result, so as to ensure that the data status between the slave node and the master node is consistent.

6. The method according to claim 5, characterized in that, The method further includes: Monitor the migration progress of historical document data. In response to determining that all target fields encrypted with the first key version have been migrated and updated to be re-encrypted with the second key version and written back to storage, and that the slave node has been synchronized to the same data state as the master node, update the key rotation status corresponding to the target field to the migration completed status, and deactivate the first key version.

7. The method according to claim 1, characterized in that, The method further includes: Before receiving the key rotation instruction, the plaintext value of the target field is stored in the document data of the document database in ciphertext form, and the corresponding query token is stored in an index structure separate from the document data, so as to decouple ciphertext storage from index retrieval.

8. The method according to claim 2, characterized in that, The first index structure and the second index structure are equality index structures that support equality matching retrieval based on query tokens. The equality index structures include hash index structures, inverted index structures, and key-value index structures.

9. A database indexing device that supports online rotation of field-level keys, characterized in that, include: The status setting module is used to respond to a key rotation instruction for a target field in the document database and set the key rotation status corresponding to the target field to a migration state; wherein, the key rotation instruction is used to instruct the conversion of the first key version corresponding to the target field to a second key version; in the migration state, the target field in newly written data is encrypted using the second key version, and the target field in the document database encrypted using the first key version is gradually migrated and updated to be re-encrypted using the second key version and then written back for storage; The query token generation module is used to, in the migration state, if an equivalence query request for the target field is received, obtain the query value in the equivalence query request, generate a first query token based on the first key version and the query value, and generate a second query token based on the second key version and the query value; The parallel query module is used to perform a search in a first index structure associated with the first key version using the first query token to obtain a first result set matching the first query token, and to perform a search in a second index structure associated with the second key version using the second query token to obtain a second result set matching the second query token; wherein the search in the first index structure and the search in the second index structure are performed in parallel. The result feedback module is used to merge and deduplicate the first result set and the second result set, and generate and feedback the final query result.

10. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

Images