Method and system for searchable storage of encrypted data

Abstract

The application provides a searchable storage method and system for encrypted data, belonging to the technical field of data security storage and retrieval. The method comprises the following steps: constructing a distributed cluster comprising a document entity storage area and an inverted index structure, configuring the documents in the cluster to store ciphertexts containing sensitive fields and bind the inverted index, and configuring the document entity storage area to exclude the entity content of the sensitive fields when storing the documents; receiving business data containing sensitive field texts, writing the texts into the inverted index after word segmentation, and generating ciphertext strings by encrypting the texts and storing the ciphertext strings in the documents; using the inverted index to match target keywords during retrieval, obtaining a list of unique identifiers of the matching documents, obtaining the ciphertext strings according to the identifiers, and filling the decrypted ciphertext strings into the retrieval results. The application realizes the physical separation of the index and the entity, supports full-text retrieval, reduces storage overhead, and improves data security.

Description

Technical Field

[0001] This application belongs to the field of data security storage and retrieval technology, specifically relating to a searchable storage method and system for encrypted data. Background Technology

[0002] With the development of cloud computing and big data, encrypting sensitive data such as personal identity information, financial data, and medical records and storing them in the cloud or distributed databases has become a common practice for ensuring data security and privacy compliance. However, while data encryption protects data confidentiality, it also fundamentally destroys the semantic features of the original plaintext, making efficient retrieval of encrypted data a long-standing and difficult problem in this field.

[0003] Currently, fuzzy retrieval of encrypted data is performed by adopting a dual-field architecture that separates the complete ciphertext from the retrieval index, combined with a sliding window word segmentation strategy. The original plaintext is segmented into words using a fixed-length sliding window to generate all possible substring combinations, which are then encrypted and stored as retrieval strings. During a query, the query terms are converted to the same word segmentation format, encrypted, and then subjected to fragment matching. The initially matched candidate results are then decrypted and filtered a second time to confirm the final result.

[0004] However, the above solution has the following problems: First, the storage overhead is enormous. To support fuzzy queries at any position, multiple substrings proportional to the text length must be generated for each document and encrypted and stored separately, resulting in a significant increase in storage space and excessively high storage costs in scenarios with hundreds of millions of data points. Second, the inverted index becomes invalid when encrypted documents are updated. After a document is updated, it must be re-encrypted and the entire inverted index must be rebuilt. The original index can no longer be used for retrieval because the correspondence between the word segmentation entries and the encrypted content changes, resulting in extremely low update efficiency and severely limited system availability. Summary of the Invention

[0005] In a first aspect, embodiments of this application provide a searchable storage method for encrypted data, comprising the following steps: S1. Construct a cluster for distributed search storage, the cluster including a document entity storage area and an inverted index structure, the inverted index structure being used to establish a mapping relationship between keywords and documents for the documents stored in the document entity storage area; S2. Configure the documents in the cluster as follows: Store the ciphertext containing the sensitive fields and bind the inverted index of the sensitive fields; For sensitive fields, configure an inverted index to retain the index entries of the sensitive fields, and configure the document entity storage area to exclude the entity content of sensitive fields when storing documents; S3. Receive business data containing sensitive field text, input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the inverted index; at the same time, use a preset encryption algorithm to encrypt the sensitive field text to generate a ciphertext string, associate the ciphertext string with the unique identifier of the current document, and store it in the document as the ciphertext of the sensitive field. S4. Receive a search request containing the target keywords, input the target keywords into the word segmenter to generate query terms, use the inverted index to match the query terms, obtain a list of unique identifiers of the matching documents and output them; S5. Based on the list of unique identifiers, retrieve the ciphertext string corresponding to each identifier from the document; S6. Decrypt the obtained ciphertext string, restore it to the plaintext of the sensitive fields, and fill the plaintext of the sensitive fields into the search results.

[0006] Secondly, embodiments of this application also provide a searchable storage system for encrypted data, comprising: The cluster building module is used to build a cluster for distributed search storage. The cluster includes a document entity storage area and an inverted index structure. The inverted index structure is used to establish a mapping relationship between keywords and documents for the documents stored in the document entity storage area. The document configuration module is used to configure documents in the cluster as follows: Store the ciphertext containing the sensitive fields and bind the inverted index of the sensitive fields; For sensitive fields, configure an inverted index to retain the index entries of the sensitive fields, and configure the document entity storage area to exclude the entity content of sensitive fields when storing documents; The writing module is used to receive business data containing sensitive field text, input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the inverted index; at the same time, it uses a preset encryption algorithm to encrypt the sensitive field text to generate a ciphertext string, and after associating the ciphertext string with the unique identifier of the current document, it stores it in the document as the ciphertext of the sensitive field. The retrieval module is used to receive retrieval requests containing target keywords, input the target keywords into the word segmenter to generate query terms, use the inverted index to match the query terms, obtain a list of unique identifiers of the matching documents and output them; The ciphertext retrieval module is used to retrieve the ciphertext string corresponding to each identifier from the document based on the list of unique identifiers; The decryption module is used to decrypt the obtained ciphertext string, restore it to the plaintext of the sensitive fields, and fill the plaintext of the sensitive fields into the search results.

[0007] As can be seen from the above technical solutions, this application has the following advantages: The searchable storage method and system for encrypted data provided in this application constructs a distributed cluster that separates the index from the physical entity. Sensitive field terms are written into the inverted index, while the encrypted ciphertext is associated with the document's unique identifier. Simultaneously, plaintext content of sensitive fields is excluded from the document entity storage area. This achieves full-text search capability while reducing storage overhead and avoiding the storage bloat problem caused by factor string encryption in traditional solutions. The plaintext of sensitive fields is cleared from memory immediately after word segmentation, without being written to disk or participating in replica synchronization, preventing data leakage risks during storage and transmission. The parent-child document structure allows child documents to be updated independently, avoiding the need for re-encryption and index rebuilding of sensitive fields due to changes in non-sensitive fields, thus improving system update efficiency and availability. During retrieval, the inverted index quickly locates the document identifier, and the ciphertext is retrieved as needed and decrypted and filled at the application layer, achieving efficient retrieval while ensuring data security. Attached Figure Description

[0008] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the searchable storage method for encrypted data according to the present invention.

[0010] Figure 2 This is a schematic diagram of the searchable storage system for encrypted data according to the present invention. Detailed Implementation

[0011] Various embodiments of this disclosure will be described more fully in the detailed steps of the searchable storage method for encrypted data described below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0012] This embodiment provides a searchable storage method for encrypted data, which separates the index from the entity, reduces storage overhead while supporting full-text search, improves data security by preventing plaintext from being written to disk, and optimizes update efficiency.

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 The diagram shows a flowchart of a searchable storage method for encrypted data in a specific embodiment. The method includes the following steps: S1. Construct a cluster for distributed search storage, the cluster including a document entity storage area and an inverted index structure, the inverted index structure being used to establish a mapping relationship between keywords and documents for the documents stored in the document entity storage area; It should be noted that the inverted index structure achieves an efficient mapping from keywords to documents, making the time complexity of retrieval operations logarithmically related to the number of index terms rather than linearly related to the total number of documents, thus providing a data structure foundation for efficient retrieval. S2. Configure the documents in the cluster as follows: Store the ciphertext containing the sensitive fields and bind the inverted index of the sensitive fields; For sensitive fields, configure an inverted index to retain the index entries of the sensitive fields, and configure the document entity storage area to exclude the entity content of sensitive fields when storing documents; It should be noted that this step separates index terms from entity content at the configuration level. That is, only lightweight search terms are retained in the inverted index, and no original content of sensitive fields is retained in the entity storage area. This makes the storage space no longer grow linearly with the number of segmented substrings. At the same time, through the configuration of separating index and entity storage, the index terms of sensitive fields are retained in the inverted index to support retrieval, while the entity content of sensitive fields is excluded when the document is persisted. The two are controlled separately through a unified configuration strategy. S3. Receive business data containing sensitive field text, input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the inverted index; at the same time, use a preset encryption algorithm to encrypt the sensitive field text to generate a ciphertext string, associate the ciphertext string with the unique identifier of the current document, and store it in the document as the ciphertext of the sensitive field. It should be noted that this step adopts a dual-track parallel writing method. Writing to the inverted index and encrypted storage are independent of each other and do not interfere with each other. The same plaintext data generates two forms (i.e., index terms and ciphertext strings) and stores them in different storage locations, providing a data foundation for retrieval and decryption. S4. Receive a search request containing the target keywords, input the target keywords into the word segmenter to generate query terms, use the inverted index to match the query terms, obtain a list of unique identifiers of the matching documents and output them; It should be noted that the retrieval process is entirely based on the inverted index, reusing the native matching capabilities and performance optimizations of the underlying search engine. There is no need to traverse the entire document, and the response speed is consistent with plaintext search. Moreover, the inverted index matches the original word segmentation entries, and the matching result is the exact document hit. There is no need for secondary filtering, which avoids the false alarm problem in traditional solutions. S5. Based on the list of unique identifiers, retrieve the ciphertext string corresponding to each identifier from the document; It should be noted that the ciphertext retrieval is only performed on the matched document identifiers, and the number of ciphertexts to be retrieved is equal to the total number of matched documents, rather than the traditional approach which requires reading and decrypting a large number of candidate sets before filtering. This reduces I / O operations and decryption computation. S6. Decrypt the obtained ciphertext string, restore it to the plaintext of the sensitive fields, and fill the plaintext of the sensitive fields into the search results; It should be noted that the decryption operation in this step is performed after the retrieval and ciphertext acquisition are completed, and only the documents matched by the retrieval are decrypted. In traditional solutions, decryption is performed before result filtering, requiring the decryption of all initially matched candidate documents and then comparison and confirmation one by one. The decryption computation covers a large number of documents that are ultimately filtered out. In this step, both retrieval and ciphertext acquisition are completed before decryption, and the number of decryption operations equals the number of documents returned in the end, avoiding the waste of decryption computation resources in traditional solutions. At the same time, decryption is physically isolated from the cluster's storage and retrieval layers, so changing the encryption algorithm or upgrading the underlying storage engine will not affect the business logic, resulting in good scalability.

[0015] This embodiment eliminates storage redundancy by separating the index from the physical entity, making the storage overhead close to that of plaintext; retrieval relies entirely on the native inverted index, avoiding false detections and missed detections, and eliminating the need for secondary filtering; plaintext is cleared immediately after word segmentation, and retrieval does not involve decryption, while the data is decrypted before display, ensuring that the data remains encrypted throughout the process.

[0016] Furthermore, as a refinement and extension of the specific implementation methods described above, and to fully illustrate the specific implementation process in this embodiment, another searchable storage method for encrypted data is provided, which includes the following steps: S1. Construct a cluster for distributed search storage, the cluster including a document entity storage area and an inverted index structure, the inverted index structure being used to establish a mapping relationship between keywords and documents for the documents stored in the document entity storage area; The specific steps of step S1 are as follows: S11. Elasticsearch is used as the underlying search engine for the cluster; S12. Configure the cluster's index mapping, setting the following parameters for sensitive fields: The indexing option is enabled, the storage option is disabled, and the document value option is disabled to instruct the cluster to create an inverted index for the sensitive field, and at the same time to instruct the cluster to exclude the entity content of the sensitive field when persisting the document. S13. Set source data exclusion parameters to exclude sensitive fields from the document's source data; The cluster uses a parent-child index structure: The documents in the cluster are divided into parent documents and child documents. The parent document stores the ciphertext containing sensitive fields and is bound to the first inverted index of the sensitive fields. The child document stores the auxiliary fields of the parent document and is bound to the second inverted index of the auxiliary fields. The child document is associated with the parent document through the unique identifier of the parent document, and the child document is updated independently of the parent document. It should be noted that a cluster refers to a distributed system containing at least one master node and multiple data nodes. In one specific embodiment, Elasticsearch version 7.6 is used as the underlying search engine of the cluster, which is bound to the Lucene 8.4.0 kernel. The document entity storage area corresponds to the _source storage and independent storage fields in Elasticsearch, and the inverted index structure corresponds to the inverted index files (.tim and .tip files) in the Lucene kernel. The inverted index structure of the Elasticsearch cluster includes a term dictionary and an inverted posting list. The term dictionary stores an ordered set of all index terms, and the inverted posting list stores a list of document IDs corresponding to each term, as well as meta-information such as term frequency and position. Together, they constitute the mapping relationship from keywords to documents. In other embodiments, the cluster can also be implemented using Apache Solr or other distributed search engines that support inverted indexes. S2. Configure the documents in the cluster as follows: Store the ciphertext containing the sensitive fields and bind the inverted index of the sensitive fields; For sensitive fields, configure an inverted index to retain the index entries of the sensitive fields, and configure the document entity storage area to exclude the entity content of sensitive fields when storing documents; The specific steps of step S2 are as follows: S21. Configure the parent document to store ciphertext containing sensitive fields, and establish a binding relationship between the parent document and the first inverted index, so that the index terms in the first inverted index can be mapped to the unique identifier of the parent document; S22. Configure the first inverted index to store only the index terms and the mapping relationship between the index terms and the unique identifier of the parent document after receiving the index terms; do not store the plaintext or ciphertext entity content of sensitive fields. S23. Configure the document entity storage area to skip the entity content of sensitive fields when persisting the parent document, so that the entity content of sensitive fields is not written to disk in the document entity storage area; S24. Configure the cluster to use distributed synchronization so that the entity content of sensitive fields is not transmitted to the data nodes where the replica shards are located during the replica synchronization process; It should be noted that the specific steps for configuring the document entity storage area to exclude the entity content of sensitive fields when storing documents are as follows: In the Elasticsearch index mapping, set "index":true for the sensitive field to indicate that an inverted index is created for the field; set "store":false and "doc_values":false to indicate that the entity content of the field is not stored during persistent storage; and set the "_source.excludes" parameter to exclude this field from the _source metadata. The combined effect of the above configuration is that during the index writing phase, the tokenized terms of the sensitive field enter the inverted index, but the original plaintext and ciphertext of the sensitive field are not written to the _source field, not written to the independent storage field, and not written to the document value doc_values ​​structure. This configuration is different from single _source filtering, which only hides the fields returned by the interface, while the original field data remains in the transaction log (Translog), shard replicas, and document entities. This step, through the combination of "store":false and "doc_values":false, achieves storage exclusion across the entire chain from the Lucene underlying layer to the cluster replicas. S3. Receive business data containing sensitive field text, input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the inverted index; at the same time, use a preset encryption algorithm to encrypt the sensitive field text to generate a ciphertext string, associate the ciphertext string with the unique identifier of the current document, and store it in the document as the ciphertext of the sensitive field. The specific steps of step S3 are as follows: S31. Receive business data containing sensitive field text, extract the sensitive field text, and assign a unique identifier to the current parent document. The unique identifier is the internal document number or external business primary key assigned by the cluster to the current parent document. S32. Input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the first inverted index; Specifically, the sensitive field text is input into a word segmenter, which segments the sensitive field text into words using a single character as the smallest cutting unit, generating at least one single-character index term. The offset of each index term is marked character by character according to the position of its corresponding character in the sensitive field text. After word segmentation is completed, the index term is written into the first inverted index. S33. After word segmentation is completed, the cluster's native memory reclamation mechanism is used to clear the original plaintext data of sensitive field text in memory, so that the plaintext of sensitive fields is not stored in the index file, not recorded in the transaction log, and not synchronized to the cluster's replica data nodes. S34. Use the SM4 symmetric encryption algorithm (Chinese national standard) to encrypt the sensitive text field, generating a ciphertext string. The encryption formula is:

[0017] in, This represents the encryption function in SM4-GCM mode. For encryption key, For sensitive fields, plaintext Given a random initial vector, The generated ciphertext string; S35. After associating the ciphertext string with a unique identifier, store it in the document as the ciphertext of a sensitive field in the parent document; It should be noted that in this step, inputting the sensitive field text into the word segmenter to generate index entries and encrypting the sensitive field text using a preset encryption algorithm to generate ciphertext strings are two independent operations executed in parallel. They share the same input sensitive field text, but their processing paths are independent and do not interfere with each other. In a specific embodiment, the word segmenter uses the native Chinese character-by-character granularity word segmenter (i.e., character segmentation mode, corresponding to Elasticsearch 7.6) that comes with the Elasticsearch 7.6. The ngram segmenter in 7.6, configured with min_gram=1 and max_gram=1, has the following segmentation logic: It uses a single Chinese character as the smallest segmentation unit, independently segmenting each valid character in the sensitive field text to generate single-character entries; during segmentation, it filters non-Chinese characters and punctuation marks. Specifically, it determines the Unicode encoding range for each character in the sensitive field text, retaining only characters belonging to the Chinese Unicode encoding range [0x4E00, 0x9FFF] as valid segmentation units, and treating characters outside this range (such as numbers, English letters, and punctuation marks) as delimiters, without generating corresponding entries; it does not perform multi-character aggregation window merging during segmentation, and each valid character generates an independent entry; let the sensitive field text be... Where L is the text length, then for each character... (where 1≤i≤L) Generate an independent term The offset is denoted as i-1. For example, for the sensitive field text "FICTIONAL", the tokenizer generates nine single-character entries: "F" (offset 0), "I" (offset 1), "C" (offset 2), "T" (offset 3), "I" (offset 4), "O" (offset 5), "N" (offset 6), "A" (offset 7), and "L" (offset 8). Each entry records its offset independently, and multi-character combination entries such as "FI" and "IC" are not generated. This tokenization method ensures that each character in any search keyword can be independently matched in the inverted index, and the relative positional relationship between characters can be determined through the offset marker. In addition, the cluster's native memory reclamation mechanism is used to clear sensitive words from memory. The original plaintext data of the segment text is as follows: After the sensitive field text is segmented and parsed and the inverted index terms are constructed, the Elasticsearch native memory reclamation mechanism is called to mark the original plaintext data as reclaimable. The original plaintext data in memory is released immediately after the write transaction is committed. At the same time, during the Elasticsearch Translog transaction log writing process, the logging policy is configured to record only the index operation metadata and not the original plaintext content of the sensitive field. Through the above mechanism, the plaintext of the sensitive field is not written to the index file, does not fall into the transaction log, and is not synchronized to the cluster replica. For the encryption process, in a specific embodiment, the SM4-GCM mode is adopted. The encryption key K is generated and securely stored by the Key Management System (KMS). The initialization vector IV is randomly generated in each encryption operation to ensure that the same plaintext generates different ciphertext strings in different encryption operations, and to avoid the ciphertext mode from leaking plaintext information. S4. Receive a search request containing the target keywords, input the target keywords into the word segmenter to generate query terms, use the inverted index to match the query terms, obtain a list of unique identifiers of the matching documents and output them; The specific steps of step S4 are as follows: S41. Receive a search request containing target keywords and extract the target keywords; the search request carries user identification and query conditions; S42. Input the target keywords into the word segmenter to generate query terms; The word segmenter segments the target keyword using a single character as the smallest cutting unit, generating at least one single-character query term. The offset of each query term is marked character by character according to the position of its corresponding character in the target keyword. S43. Use the first inverted index to match the query terms, match each query term with the index terms stored in the inverted index one by one, and filter out all parent documents that contain at least one query term; S44. Based on the matching results, obtain a list of unique identifiers for the matching parent documents and output it to the application layer; S45. When a retrieval request is initiated after a child document is updated, the retrieval conditions are applied to both the first inverted index of the parent document and the second inverted index of the child document simultaneously, and the retrieval process is not affected by the update of the child document. It should be noted that the retrieval process in this step is entirely based on dictionary matching of the inverted index and does not involve decryption of the encrypted data. In a specific embodiment, the word segmenter segments the target keyword using the same rules as S3, using a single character as the smallest segmentation unit to generate at least one single-character query term. The offset of each query term is marked character by character according to the position of its corresponding character in the target keyword. During retrieval, each query term is matched one by one with the index terms stored in the inverted index to filter out all documents containing at least one query term. For example, if the target keyword is "AL", after word segmentation, query terms "A" (offset 0) and "L" (offset 1) are generated. The inverted index matches all documents containing "A" or "L" and returns the union of the two results. When it is necessary to accurately match the continuous two-character phrase "AL", the offset information of "A" and "L" in each document can be used for secondary filtering in subsequent processing to retain only documents whose offsets differ by 1. Since the matching process of the inverted index is based on dictionary lookup in the Lucene kernel, the retrieval process is performed by the word segmenter. The lookup method is used, and the time complexity is logarithmic with the total number of indexed terms (O(logN)), which is much lower than the full table scan (O(N)) of the traditional LIKE '%...%' method. The unique identifier list is a set of IDs of the matching documents. This set is stored in the cluster memory in the data structure of Bitmap or SkipList and output to the application layer for use in subsequent steps. S5. Based on the list of unique identifiers, retrieve the ciphertext string corresponding to each identifier from the document; The specific steps of step S5 are as follows: S51. Receive a list of unique identifiers at the application layer. ,in This is a unique identifier for the i-th document. The total number of parent documents to be matched; S52. Construct a batch query request at the application layer, wherein the batch query request contains all identifiers in the unique identifier list; S53. At the application layer, send batch query requests to the cluster, instructing the cluster to search for the ciphertext string corresponding to each unique identifier in the parent document based on the list of unique identifiers; S54. The cluster reads the ciphertext string from the sensitive field of the corresponding parent document for each unique identifier in the unique identifier list and returns a list of ciphertext strings. and its corresponding list of initial vectors ,in It is a unique identifier for a document. Ciphertext stored in association It is the corresponding encryption initialization vector; It should be noted that, in this step, obtaining the ciphertext string corresponding to each identifier from the document based on the unique identifier list specifically involves the following steps: After receiving the unique identifier list output by S4, the application layer constructs a batch query request containing all document IDs in the identifier list, and then sends the batch query request to the cluster. Upon receiving the request, the cluster reads the ciphertext string and the corresponding initial vector from the sensitive field of the corresponding parent document for each document ID. Since the document entity storage area has been configured in S2 to exclude the entity content of sensitive fields, the ciphertext string and initial vector returned by the cluster are all the contents of the sensitive fields stored in the document. The ciphertext string and initial vector are returned to the application layer through the cluster's retrieval API. All data transmitted during this process is in ciphertext form and does not contain any plaintext information. Even if the transmission link is intercepted, business data will not be leaked. S6. Decrypt the obtained ciphertext string, restore it to the plaintext of the sensitive fields, and fill the plaintext of the sensitive fields into the search results; The specific steps of step S6 are as follows: S61. Receive the list of ciphertext strings at the application layer. and the corresponding list of initial vectors ; S62. Obtain the encryption key at the application layer. Corresponding decryption key The decryption key is securely stored and provided by the key management system; S63. At the application layer, process each ciphertext... Perform the decryption operation; the decryption formula is:

[0018] in, This represents the decryption function for SM4-GCM mode. To decrypt the key, For the first A ciphertext string, For the first The initial vector corresponding to each ciphertext. To restore the first The sensitive field is in plaintext. The total number of documents matched; S64. Translate each sensitive field to plaintext at the application layer. The results are populated into the corresponding parent document to form the final search results, which are then returned to the user. It should be noted that the decryption operation in this step is performed entirely at the application layer; after receiving the ciphertext string list and the corresponding initialization vector list returned in step S5, the application layer obtains the decryption key K corresponding to the encryption key K from the key management system. - ¹; For each ciphertext C i and its corresponding initial vector IV i The application layer performs the decryption operation independently; in one specific embodiment, the decryption key K - ¹The encryption key is the same as K (SM4 is a symmetric encryption algorithm); after decryption, the plaintext P of each sensitive field is... i The data is populated into its corresponding document and assembled together with non-sensitive fields in the document (such as business codes, data entry time, data classification tags, etc.) to form a complete business response result, which is then returned to the user. This process only involves decryption calculations at the application layer, and the cluster layer does not handle plaintext at all. In the above process, the plaintext of sensitive fields is not written to persistent storage media. The retrieval process is completed in encrypted state, and the results are only decrypted and restored to plaintext at the application layer when they are finally displayed to the user.

[0019] In some embodiments, the step of configuring the encryption policy mapping is also included: In the configuration table, a set of policy parameters are dynamically specified for each sensitive field. The policy parameters include separation mode flag, encryption algorithm flag, and word segmenter parameters. The separation mode flag indicates whether the index and entity separation mode is enabled for this sensitive field; The encryption algorithm flag specifies the preset encryption algorithm used to encrypt this sensitive field; The tokenizer parameters specify the type and parameters of the tokenizer; The system reads the configuration table and performs differentiated write and retrieval processing on different sensitive fields.

[0020] For example, the "employee ID number" field in the enterprise information management system is used as a sensitive field to fully illustrate the technical solution of this application. It should be noted that the employee information used in this example is fictitious data and is only used to illustrate the technical solution of this application, and does not refer to any real individual.

[0021] If the business data contains an employee information record, the plaintext of its ID number field is P="FICTIONAL_ID_001", and the unique identifier of the document is DocID=1001.

[0022] During the writing phase, the system inputs P="FICTIONAL_ID_001" into the word segmenter. The word segmenter segments words using a single character as the smallest unit, generating index entries for each valid character, specifically including: Letter segmentation character by character: "F" (offset 0), "I" (offset 1), "C" (offset 2), "T" (offset 3), "I" (offset 4), "O" (offset 5), "N" (offset 6), "A" (offset 7), "L" (offset 8); The underscore "_" is treated as a separator and does not generate entries. Subsequent parts: "I" (offset 10), "D" (offset 11), "_" (separator, no entry generated), "0" (offset 13), "0" (offset 14), "1" (offset 15).

[0023] The system writes all the single-character index terms into the inverted index, mapping each term to document ID=1001 and recording the corresponding offset. After word segmentation is complete, the original plaintext "FICTIONAL_ID_001" in memory is immediately cleared.

[0024] Meanwhile, the system uses the SM4-GCM algorithm to encrypt P. Specifically: the system obtains the encryption key K and the randomly generated initialization vector IV, and performs the encryption operation. The ciphertext string C is generated. After associating the ciphertext string C, the initialization vector IV, and document ID=1001, it is stored as the ciphertext of the ID number field in the parent document within the cluster. The inverted index only retains each single-character term and its mapping relationship and offset to document ID=1001; it does not contain any information from the plaintext P or the ciphertext C.

[0025] If the employee's phone number field needs to be updated subsequently, since the phone number is a frequently changing auxiliary field, the update operation only involves the child document. The system writes the new value of the phone number to the child document, which is associated with the parent document via ID=1001. During the update of the child document, the parent document's ID number field and its bound first inverted index are completely unaffected, and there is no need to rebuild the parent document's inverted index due to the phone number update.

[0026] During the retrieval phase: The user inputs the target keyword Q="ID" to perform a search. The system inputs Q="ID" into the word segmenter, generating two single-character query terms: "I" (offset 0) and "D" (offset 1). The retrieval module uses the first inverted index to match the above query terms, searching for document IDs containing "I" or "D" one by one, obtaining the union of all matching document IDs, and then using the offset information of "I" and "D" in each document to perform position verification, retaining only documents where the offset of "I" differs from the offset of "D" by 1, and finally determining document ID=1001 as the matching result. The system outputs the document ID list

[1001] to the application layer.

[0027] The application layer constructs a batch query request containing the identifier 1001 and sends it to the cluster, instructing the cluster to read the ciphertext string C and initialization vector IV corresponding to ID=1001 from the parent document. The cluster then searches for the sensitive field in the corresponding parent document based on document ID=1001 and returns the ciphertext string C and the corresponding initialization vector IV to the application layer.

[0028] The application layer obtains the decryption key K corresponding to the encryption key K from the key management system. - ¹(SM4 is a symmetric encryption, K) - ¹=K), perform decryption operation on ciphertext C: The process involves restoring the plaintext of the sensitive field "FICTIONAL_ID_001". This plaintext is then used to populate the search results, assembling them together with non-sensitive metadata (such as employee name, department, and data entry time) to form a complete business response, which is then returned to the user.

[0029] The above example of the complete data flow demonstrates that: the plaintext of sensitive fields exists only momentarily in memory during the word segmentation process, and is cleared after segmentation is complete; it is not written to the index file, not recorded in the transaction log, and not synchronized to the cluster replica. The inverted index only stores single-character terms and their mapping relationship and offset to the document ID, making it impossible to reconstruct the complete original text of sensitive fields from the index. The ciphertext is stored in the parent document but cannot be directly recognized; even if the cluster is compromised, the attacker only obtains the ciphertext. The application layer only retrieves the ciphertext and performs decryption in the final stage of a search hit. The decrypted plaintext is directly used for result presentation and is not written back to the cluster. Throughout the process, the plaintext of sensitive fields is cleared from memory after word segmentation and is not written to any persistent storage medium. The cluster only retains single-character terms used for retrieval and their mapping relationships. After a search hit, the application layer retrieves the ciphertext from the parent document and decrypts it to reconstruct the plaintext, thus achieving separation of the retrieval, storage, and decryption stages.

[0030] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0031] This application utilizes four collaborative features: separation of index and physical entity, splitting of parent and child documents, dual-track parallel writing, and ID-based post-decryption retrieval. First, the separation of index and physical entity eliminates redundant storage of massive quantum strings in traditional word segmentation schemes, reducing storage overhead from O(N) to O(1). Second, splitting of parent and child documents solves the problem of inverted index failure during encrypted document updates, reducing the impact of update operations from full index reconstruction to partial updates of child documents. Third, dual-track parallel writing ensures decoupling of index terms and ciphertext at the writing source; after word segmentation, the ciphertext is cleared from memory, and the plaintext is not written to the disk. Disk-based, not recorded in transaction logs, and not synchronized to cluster replicas; ID-based post-decryption retrieval ensures that decryption operations are only performed on the final hit document, avoiding the computational waste of decrypting a large number of documents before filtering in traditional solutions; the above four features work together: without separation of indexes and entities, storage redundancy cannot be eliminated; without parent and child document splitting, retrieval stability in high-frequency update scenarios cannot be guaranteed; without dual-track parallel writing, plaintext may remain in transaction logs or replicas, resulting in incomplete data privacy protection; without post-decryption retrieval, query performance is still limited by invalid decryption overhead.

[0032] like Figure 2 As shown, the following are embodiments of the searchable storage system for encrypted data provided in this disclosure. This system and the searchable storage method for encrypted data in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the searchable storage system for encrypted data, please refer to the embodiments of the searchable storage method for encrypted data described above.

[0033] The system includes: The cluster building module is used to build a cluster for distributed search storage. The cluster includes a document entity storage area and an inverted index structure. The inverted index structure is used to establish a mapping relationship between keywords and documents for the documents stored in the document entity storage area. The document configuration module is used to configure documents in the cluster as follows: Store the ciphertext containing the sensitive fields and bind the inverted index of the sensitive fields; For sensitive fields, configure an inverted index to retain the index entries of the sensitive fields, and configure the document entity storage area to exclude the entity content of sensitive fields when storing documents; The writing module is used to receive business data containing sensitive field text, input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the inverted index; at the same time, it uses a preset encryption algorithm to encrypt the sensitive field text to generate a ciphertext string, and after associating the ciphertext string with the unique identifier of the current document, it stores it in the document as the ciphertext of the sensitive field. The retrieval module is used to receive retrieval requests containing target keywords, input the target keywords into the word segmenter to generate query terms, use the inverted index to match the query terms, obtain a list of unique identifiers of the matching documents and output them; The ciphertext retrieval module is used to retrieve the ciphertext string corresponding to each identifier from the document based on the list of unique identifiers; The decryption module is used to decrypt the obtained ciphertext string, restore it to the plaintext of the sensitive fields, and fill the plaintext of the sensitive fields into the search results.

[0034] This embodiment achieves efficient full-text retrieval of encrypted data and reduces storage overhead through the interactive collaboration of the cluster construction module, document configuration module, writing module, retrieval module, encrypted text acquisition module, and decryption module, and through an index and entity separation architecture; it also ensures that sensitive plaintext is not written to disk or transmitted, thereby improving data security.

[0035] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for searchable storage of encrypted data, characterized in that, Includes the following steps: S1. Construct a cluster for distributed search storage, the cluster including a document entity storage area and an inverted index structure, the inverted index structure being used to establish a mapping relationship between keywords and documents for the documents stored in the document entity storage area; S2. Configure the documents in the cluster as follows: Store the ciphertext containing the sensitive fields and bind the inverted index of the sensitive fields; For sensitive fields, configure an inverted index to retain the index entries of the sensitive fields, and configure the document entity storage area to exclude the entity content of sensitive fields when storing documents; S3. Receive business data containing sensitive field text, input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the inverted index; at the same time, use a preset encryption algorithm to encrypt the sensitive field text to generate a ciphertext string, associate the ciphertext string with the unique identifier of the current document, and store it in the document as the ciphertext of the sensitive field. S4. Receive a search request containing the target keywords, input the target keywords into the word segmenter to generate query terms, use the inverted index to match the query terms, obtain a list of unique identifiers of the matching documents and output them; S5. Based on the list of unique identifiers, retrieve the ciphertext string corresponding to each identifier from the document; S6. Decrypt the obtained ciphertext string, restore it to the plaintext of the sensitive fields, and fill the plaintext of the sensitive fields into the search results.

2. The searchable storage method for encrypted data according to claim 1, characterized in that, The specific steps of step S1 are as follows: S11. Elasticsearch is used as the underlying search engine for the cluster; S12. Configure the cluster's index mapping, setting the following parameters for sensitive fields: The indexing option is enabled, the storage option is disabled, and the document value option is disabled to instruct the cluster to create an inverted index for the sensitive field, and at the same time to instruct the cluster to exclude the entity content of the sensitive field when persisting the document. S13. Set source data exclusion parameters to exclude sensitive fields from the document's source data.

3. The method for searchable storage of encrypted data according to claim 1, characterized in that, The cluster uses a parent-child index structure: Documents in the cluster are divided into parent documents and child documents. The parent document stores the ciphertext containing sensitive fields and is bound to the first inverted index of the sensitive fields. The child document stores the auxiliary fields of the parent document and is bound to the second inverted index of the auxiliary fields. The child document is associated with the parent document through the unique identifier of the parent document, and the child document is updated independently of the parent document.

4. The searchable storage method for encrypted data according to claim 3, characterized in that, The specific steps of step S2 are as follows: S21. Configure the parent document to store ciphertext containing sensitive fields and establish the binding relationship between the parent document and the first inverted index; S22. Configure the first inverted index to store only the index terms and the mapping relationship between the index terms and the unique identifier of the parent document after receiving the index terms; do not store the plaintext or ciphertext entity content of sensitive fields. S23. Configure the document entity storage area to skip the entity content of sensitive fields when persisting the parent document, so that the entity content of sensitive fields is not written to disk in the document entity storage area; S24. Configure the cluster to use distributed synchronization so that the entity content of sensitive fields is not transmitted to the data nodes where the replica shards are located during the replica synchronization process.

5. The searchable storage method for encrypted data according to claim 3, characterized in that, The specific steps of step S3 are as follows: S31. Receive business data containing sensitive field text, extract the sensitive field text, and assign a unique identifier to the current parent document; S32. Input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the first inverted index; S33. After word segmentation is completed, the cluster's native memory reclamation mechanism is used to clear the original plaintext data of sensitive field text in memory, so that the plaintext of sensitive fields is not stored in the index file, not recorded in the transaction log, and not synchronized to the cluster's replica data nodes. S34. Use the SM4 symmetric encryption algorithm (Chinese national standard) to encrypt the sensitive text field, generating a ciphertext string. The encryption formula is: in, This represents the encryption function in SM4-GCM mode. For encryption key, For sensitive fields, plaintext Given a random initial vector, The generated ciphertext string; S35. After associating the ciphertext string with a unique identifier, store it in the document as the ciphertext of a sensitive field in the parent document.

6. The searchable storage method for encrypted data according to claim 3, characterized in that, The specific steps of step S4 are as follows: S41. Receive a search request containing target keywords and extract the target keywords; the search request carries user identification and query conditions; S42. Input the target keywords into the word segmenter to generate query terms; S43. Use the first inverted index to match the query terms, match each query term with the index terms stored in the inverted index one by one, and filter out all parent documents that contain at least one query term; S44. Based on the matching results, obtain a list of unique identifiers for the matching parent documents and output it to the application layer; S45. When a retrieval request is initiated after a child document is updated, the retrieval conditions apply simultaneously to the first inverted index of the parent document and the second inverted index of the child document, and the retrieval process is not affected by the update of the child document.

7. The searchable storage method for encrypted data according to claim 6, characterized in that, The specific steps of step S5 are as follows: S51. Receive a list of unique identifiers at the application layer. ,in This is a unique identifier for the i-th document. The total number of parent documents to be matched; S52. Construct a batch query request at the application layer, wherein the batch query request contains all identifiers in the unique identifier list; S53. At the application layer, send batch query requests to the cluster, instructing the cluster to search for the ciphertext string corresponding to each unique identifier in the parent document based on the list of unique identifiers; S54. The cluster reads the ciphertext string from the sensitive field of the corresponding parent document for each unique identifier in the unique identifier list and returns a list of ciphertext strings. and its corresponding list of initial vectors ,in It is a unique identifier for a document. Ciphertext stored in association It is the corresponding encryption initialization vector.

8. The method for searchable storage of encrypted data according to claim 7, characterized in that, The specific steps of step S6 are as follows: S61. Receive the list of ciphertext strings at the application layer. and the corresponding list of initial vectors ; S62. Obtain the encryption key at the application layer. Corresponding decryption key The decryption key is securely stored and provided by the key management system; S63. At the application layer, process each ciphertext... Perform the decryption operation; the decryption formula is: in, This represents the decryption function for SM4-GCM mode. To decrypt the key, For the first A ciphertext string, For the first The initial vector corresponding to each ciphertext. To restore the first The sensitive fields are in plaintext. The total number of documents matched; S64. Translate each sensitive field to plaintext at the application layer. The results are populated into the corresponding parent document to form the final search results, which are then returned to the user.

9. The method for searchable storage of encrypted data according to claim 1, characterized in that, It also includes the step of configuring the encryption policy mapping: In the configuration table, a set of policy parameters are dynamically specified for each sensitive field. The policy parameters include separation mode flag, encryption algorithm flag, and word segmenter parameters. The separation mode flag indicates whether the index and entity separation mode is enabled for this sensitive field; The encryption algorithm flag specifies the preset encryption algorithm used to encrypt this sensitive field; The tokenizer parameters specify the type and parameters of the tokenizer; The system reads the configuration table and performs differentiated write and retrieval processing on different sensitive fields.

10. A searchable storage system for encrypted data, characterized in that, include: The cluster building module is used to build a cluster for distributed search storage. The cluster includes a document entity storage area and an inverted index structure. The inverted index structure is used to establish a mapping relationship between keywords and documents for the documents stored in the document entity storage area. The document configuration module is used to configure documents in the cluster as follows: Store the ciphertext containing the sensitive fields and bind the inverted index of the sensitive fields; For sensitive fields, configure an inverted index to retain the index entries of the sensitive fields, and configure the document entity storage area to exclude the entity content of sensitive fields when storing documents; The writing module is used to receive business data containing sensitive field text, input the sensitive field text into the word segmenter to generate index terms, and write the index terms into the inverted index; at the same time, it uses a preset encryption algorithm to encrypt the sensitive field text to generate a ciphertext string, and after associating the ciphertext string with the unique identifier of the current document, it stores it in the document as the ciphertext of the sensitive field. The retrieval module is used to receive retrieval requests containing target keywords, input the target keywords into the word segmenter to generate query terms, use the inverted index to match the query terms, obtain a list of unique identifiers of the matching documents and output them; The ciphertext retrieval module is used to retrieve the ciphertext string corresponding to each identifier from the document based on the list of unique identifiers; The decryption module is used to decrypt the obtained ciphertext string, restore it to the plaintext of the sensitive fields, and fill the plaintext of the sensitive fields into the search results.

Images