A vulnerability knowledge base full-text retrieval method based on encrypted document analysis and secure index construction

Abstract

The application discloses a vulnerability knowledge base full-text retrieval method based on encrypted document analysis and secure index construction, which is characterized in that: under the premise that the document is stored in an encrypted form, the encrypted document is temporarily decrypted in memory and the text content is extracted by calling a decryption service; a full-text index is constructed based on the document metadata and the text content, and the index data is stored in an encrypted storage volume to realize disk-level ciphertext storage; during retrieval, joint matching is performed on the title and the text field, and deleted documents are filtered, the document identifier is looked up according to the matched document to associate the metadata, and the result is returned; meanwhile, the index synchronization and the timing consistency check repair mechanism are triggered by document addition, deletion and modification, so that the dynamic consistency of the business data and the index data is ensured. The application realizes the full-text retrieval capability of the encrypted vulnerability document, and takes into account the retrieval efficiency, data security and system maintainability under the premise of avoiding plaintext storage and index plaintext storage.

Description

Technical Field

[0001] This application belongs to the technical field of vulnerability management and information retrieval technology, specifically involving a full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction. Background Technology

[0002] With the continuous application of vulnerability management systems in software security, vulnerability remediation, and security operations, these systems typically accumulate a large amount of vulnerability-related documentation, such as vulnerability reports, vulnerability analysis documents, remediation documentation, test verification documents, and related attachments. Current vulnerability management systems primarily rely on title search, tag search, category search, or database field search for these documents. These methods are relatively simple to implement and can meet basic information location needs. However, when users want to search based on vulnerability numbers, vulnerability characteristics, impact scope, remediation suggestions, key log snippets, or technical descriptions within the document text, relying solely on title or structured field searches often fails to accurately return the desired documents, resulting in low knowledge retrieval efficiency and undermining the full value of document content. Therefore, introducing full-text search capabilities based on the main text content, building upon existing title search capabilities, has become an important direction for improving vulnerability knowledge retrieval efficiency.

[0003] To improve retrieval capabilities, existing technologies typically employ full-text search. This involves first parsing the document content to extract plain text information, and then building a search index for the document title and body text based on a full-text indexing engine. This index is then used for word segmentation matching and relevance ranking upon receiving user keywords. However, in vulnerability management systems, relevant documents are often highly sensitive and are typically stored in encrypted form on object storage or other secure storage media. In such scenarios, the original document content exists in ciphertext, making direct text parsing and full-text index construction impossible. Therefore, to achieve full-text search, it is usually necessary to first obtain the encrypted document, then call a key management service to decrypt it, and finally parse the format of the decrypted document content to extract the body text suitable for index construction.

[0004] However, existing technologies still have many shortcomings in the scenario of full-text retrieval of encrypted documents. First, if the plaintext document is directly written to disk or cached as a temporary file after decryption, it is easy to cause sensitive data leakage, posing a significant security risk. Second, even if the full-text index is built, if the full-text index itself is stored in plaintext, the index file still contains document titles and body content, which may still create new leakage risks. Third, some existing retrieval solutions only focus on the retrieval process itself and lack efficient linkage with document business metadata in the business database. This results in multiple additional queries after a document is found, affecting return efficiency and system integration. In addition, if the index is not updated in a timely manner after a document is added, modified, logically deleted, or physically deleted, inaccurate search results, deleted documents still being detected, or new documents not being detected may occur, affecting the consistency and reliability of vulnerability knowledge base retrieval results.

[0005] Therefore, how to achieve parsing and full-text retrieval of the main text of encrypted and stored vulnerable documents while ensuring document data security, how to avoid the leakage risks brought about by decrypting plaintext and full-text indexing, and at the same time ensure the consistency between retrieval results and business document data, have become the technical problems that urgently need to be solved in the current vulnerability knowledge base retrieval technology. Summary of the Invention

[0006] To address the issues of plaintext processing leakage risks, lack of security guarantees for index storage, and difficulty in maintaining consistency between search results and business document status in existing encrypted document full-text search solutions, this application aims to provide a vulnerability knowledge base full-text search method that balances the security of dual storage of document data and index data, supports text content retrieval, and dynamically synchronizes the index with business data.

[0007] To achieve the above technical objectives, this application specifically adopts the following technical solution: One aspect of this application provides a full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction, comprising the following steps: S1. Obtain the business metadata and encrypted document of the document to be processed: Read the business metadata of the document from the business database, and obtain the corresponding encrypted document data from the encrypted document storage system according to the file path in the business metadata; S2. Temporarily decrypt the encrypted document and extract the main text content: Call the decryption service to decrypt the encrypted document data and generate temporary plaintext data that is only temporarily stored in memory; call the document parsing module to extract the text from the temporary plaintext data to obtain the main text content of the document, and release or destroy the temporary plaintext data after the extraction is completed. S3. Construct a full-text index based on document business metadata and text content: Based on the business metadata and the text content, establish a corresponding full-text index record for the document and write the full-text index record into the full-text search engine; wherein, the full-text index record includes at least a document unique identifier field, a document title field, a document text content field, and a deletion flag field; S4. Securely store the full-text index: Store the index data of the full-text search engine in an encrypted storage volume so that the index data is saved in encrypted form at the disk layer; S5. Receive user search request and perform full-text search: Receive keywords input by user, construct a joint search statement for document title field and document body content field, filter deleted documents in combination with the deletion mark field, and then perform search in the full-text search engine to obtain the document identifier set of the hit documents. S6. Associate business metadata and return search results: Based on the document identifier set, query the business database to obtain the business metadata of the hit document, and then combine the business metadata with the search results and return it. S7. Perform document and index synchronization: When a document is added, modified, logically deleted, or physically deleted, the corresponding index synchronization process is triggered to maintain the consistency between business document data and full-text index data.

[0008] In one implementation, the business metadata in step S1 includes a unique document identifier, document title, file path, update time, and deletion status.

[0009] In one implementation, the document parsing module in step S2 is adapted to extract text from multiple document formats, including txt, pdf, doc, and docx.

[0010] In one implementation, the full-text index record in step S3 further includes a file path field and an update time field; wherein the document title field and the document body content field serve as retrieval fields, and the update time field serves as a sorting field.

[0011] In one implementation, step S5 further includes: verifying the user's search permissions before performing the search, and constructing the combined search statement after the verification is passed.

[0012] In one implementation, the index synchronization process in step S7 includes: When a document is added, a new full-text index record is created; When a document modification operation occurs, update the text content and update time in the corresponding full-text index record; When a document is logically deleted, the deletion flag field in the corresponding full-text index record will be updated to indicate that it has been deleted. When a document is physically deleted, the corresponding full-text index record is deleted.

[0013] In one implementation, the index synchronization process in step S7 is triggered directly through the business interface or through database change monitoring.

[0014] In one implementation, the method further includes the step of performing index consistency verification and repair: by periodically verifying the document data in the business database and the index data in the full-text search engine, repairing situations where there are missing indexes, redundant indexes, or inconsistent deletion statuses.

[0015] In one implementation, when index synchronization fails, the synchronization task is retried via a retry mechanism.

[0016] In one implementation, the encrypted document storage system, decryption service, document parsing module, full-text search engine, and business database are deployed in the same internal secure network environment.

[0017] Compared with existing vulnerability knowledge retrieval schemes that only support title retrieval or structured field retrieval, the full-text retrieval method for vulnerability knowledge bases based on encrypted document parsing and security indexing provided in this application has at least the following advantages: 1. Implement full-text search capabilities in encrypted document storage scenarios.

[0018] This application enables the parsing and full-text indexing of document content even when documents are stored in encrypted form, expanding the system from simple title-based retrieval to a combined title and content-based retrieval. Compared to methods relying solely on document titles, tags, or database fields, this application supports content retrieval based on vulnerability numbers, vulnerability descriptions, impact scope, remediation suggestions, and other keywords within the text, thereby improving the ability to locate vulnerability knowledge documents and increasing the retrieval hit rate.

[0019] 2. Reduced the risk of plaintext leakage during processing.

[0020] This application, during index construction, temporarily decrypts encrypted documents and only stores the plaintext data in memory for parsing, without writing the plaintext data to local disks or other persistent media. The temporary plaintext data is also promptly released or destroyed after the main text extraction is complete. Simultaneously, the logs during the decryption process are anonymized, and no plaintext content is recorded. Therefore, this application can effectively reduce the risk of exposing decrypted plaintext in the processing chain while simultaneously constructing a full-text index, thus improving the security of vulnerable document processing.

[0021] 3. Enhanced the storage security of the full-text index data itself.

[0022] This application not only encrypts and protects the original document, but also stores the full-text index data in an encrypted storage volume, ensuring that the index file exists in encrypted form at the disk level. Since the full-text index contains document titles, body text, and related search information, storing the index file in plaintext could also lead to the leakage of sensitive information. This application avoids the problem of directly exposing the index content by encrypting the index data, thus forming a dual protection mechanism for both the original document and the search index.

[0023] 4. Improved the consistency between search results and business data, and enhanced system maintainability.

[0024] This application triggers corresponding index synchronization operations when documents are added, modified, logically deleted, or physically deleted. It also maintains consistency between document data and full-text index data in the business database through failure retry and timed verification and repair mechanisms. Compared to existing solutions that only focus on index creation while neglecting subsequent synchronization and maintenance, this application reduces problems such as new documents not being indexed, deleted documents still being detected, and index content not being updated in a timely manner, thereby improving the accuracy, stability, and maintainability of full-text search results and system operation.

[0025] 5. Improved the overall usability of the vulnerability knowledge base retrieval system.

[0026] After completing the full-text search, this application can retrieve business metadata such as document title, file size, update time, and file path from the business database based on the matched document identifier, and assemble it into a results list for front-end display. Therefore, this application can not only complete text matching, but also link the search results with business management information, improving the readability and practical value of the search results, and is more suitable for knowledge retrieval, document location, and security operation and maintenance support scenarios in vulnerability management systems.

[0027] In summary, this application achieves full-text retrieval of vulnerability knowledge documents while ensuring secure storage of encrypted documents. It also takes into account the security of plaintext processing, secure index storage, accurate retrieval results, and consistent index data, and has strong practical value and promotional significance. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall architecture of the vulnerability knowledge base full-text retrieval method described in this application. Detailed Implementation

[0029] The technical solution of this application will be clearly and completely described below with reference to specific embodiments. However, those skilled in the art will understand that the embodiments described below are only some embodiments of this application, not all embodiments, and are only used to illustrate this application, and should not be regarded as limiting the scope of this application. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] To address the problems in existing technologies, such as the inability to directly perform full-text searches on encrypted vulnerable documents, the risk of leakage of decrypted plaintext data, the inherent vulnerability of plaintext exposure in full-text indexes, and the difficulty in ensuring consistency between document data and index data, this application provides a full-text search method for vulnerability knowledge bases based on encrypted document parsing and secure index construction. This method, building upon existing title-based retrieval, introduces the ability to search the document's main text content. It constructs a complete processing chain encompassing encrypted document acquisition, temporary decryption, text parsing, full-text index construction, full-text search execution, result association and return, and index synchronization and consistency assurance, thereby achieving secure full-text search for vulnerability knowledge base scenarios.

[0031] This application provides a full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security indexing, including the following steps: S1. Obtain the business metadata and encrypted document data of the document to be processed. 1) Business metadata reading After the processing flow is initiated, the executing entity first performs a data query operation on the business database. The target record for the query is the business metadata corresponding to the document to be processed. The business metadata fields read at least include the document's unique identifier, document title and description, file path indicating the actual storage location of the encrypted document, update time information recording the last change time of the document content, and a deletion status flag indicating whether the document is currently in a logically deleted state.

[0032] 2) Obtaining encrypted document data After completing the business metadata reading operation, a document data retrieval request is initiated to the encrypted document storage system based on the file path information in the obtained business metadata record. The encrypted document storage system responds to this request, locates the encrypted document data uniquely corresponding to that file path in its storage space, and returns the encrypted document data. This encrypted document data is in ciphertext form after being processed by an encryption algorithm; it does not reveal the readable original text content without the decryption service.

[0033] S2. Perform temporary decryption of the encrypted document and extract the main text content. After the encrypted document data transmitted in step S1 arrives at the decryption service, the decryption service initiates a key acquisition request to the key management service. The key management service responds to this request and returns decryption key information corresponding to the encrypted document data. The decryption service uses the acquired decryption key information to perform decryption operations on the encrypted document data, and the output of the decryption operation is temporary plaintext data. It is understood that this temporary plaintext data, after being generated, is stored in the memory area of ​​the executing entity and is not written to local disk storage media or other persistent storage devices.

[0034] After the decryption service generates temporary plaintext data, it calls the document parsing module to process it. The document parsing module first detects the data structure features or header information of the temporary plaintext data to identify the specific document format type. After determining the document format type, the document parsing module calls the text extraction logic matching that format type to extract the text information representing the main content of the document from the temporary plaintext data; this text information is the document's main text content. The document parsing module has the capability to handle text extraction tasks for various document formats, and its compatible document format types include plain text, portable document formats, traditional binary composite document formats, and modern composite document formats based on open packaging conventions.

[0035] After the document parsing module completes the extraction of the main text content and returns the extracted main text content to the caller, the executing entity performs a destruction or release operation on the temporary plaintext data stored in the memory area. The destruction or release operation marks the memory area occupied by the temporary plaintext data as overwriteable or reallocatable, and the temporary plaintext data itself has no possibility of being read or recovered again.

[0036] S3. Construct a full-text index based on document business metadata and text content. The process involves obtaining the business metadata acquired in step S1 and the main text content extracted in step S2. The business metadata includes a unique document identifier, document title, file path, update time, and deletion status. The main text content provides the body text of the document to be retrieved.

[0037] After obtaining the business metadata and the main text content, a full-text index record uniquely corresponding to the currently processed document is constructed. This full-text index record is a set of preset fields, and the values ​​of each field are filled by the corresponding items in the business metadata and the main text content.

[0038] The full-text index record contains the following fields, and their field names, content sources, and uses in the retrieval process are shown in Table 1 below: Table 1 Index Structure Design

[0039] The completed full-text index record is written to the index storage structure managed by the full-text search engine. After the write operation is executed, the text information contained in the document and the associated business metadata attributes are included in the searchable scope of the full-text search engine.

[0040] S4. Securely store the full-text index. After the constructed full-text index records are written into the full-text search engine in step S3, the index data of the full-text search engine is stored in an encrypted storage volume so that the index data is saved in encrypted form at the disk layer.

[0041] Specifically, the implementation process of step S4 is as follows: When the full-text search engine performs the write operation of index records, its underlying storage path is configured to point to a pre-created encrypted storage volume. The encrypted storage volume is a type of logical storage unit with data encryption capabilities. After receiving the data to be written, but before actually submitting the data to the physical disk device, it calls the encryption module to perform encryption transformation processing on the data content to be written.

[0042] When the full-text search engine writes index data containing document title, document body content, and deletion marker fields to the encrypted storage volume, the index data, after undergoing the encryption transformation, is persistently stored in ciphertext form at the disk layer. At the disk physical sector or file system level, the stored index data appears as a ciphertext sequence that cannot be directly read or parsed, thus preventing unauthorized access to the document text information carried by the index data when the storage medium is removed from the operating environment.

[0043] S5. Receive user search requests and perform full-text search. After the secure storage of the full-text index data is completed in step S4, the document content query request initiated by the user is responded to in step S5. The specific implementation process of step S5 is as follows: 1) Receiving user search requests.

[0044] The backend service program receives a search request from the user, which contains at least one or more search keywords entered by the user. These search keywords represent the text fragments the user expects to find in the document title or the document body.

[0045] 2) Verification of user search permissions Before performing a substantive search operation based on the search keywords, the backend service program first obtains the user's identity information that initiated the current search request, and verifies whether the user has the permission to perform a full-text document search operation according to the preset permission determination rules.

[0046] The permission determination rules can be based on user roles, the user's organization, or the access control list the user possesses. Subsequent retrieval processing will only be triggered if the verification result indicates the user has the corresponding retrieval permissions; if the verification result indicates the user does not have the corresponding retrieval permissions, the processing of the current retrieval request will be terminated and a "no permission" message will be returned.

[0047] 3) Construction of combined search statements After successful verification, the backend service program constructs a combined search statement for the full-text search engine. The construction rules for the combined search statement are as follows: the document title field and the document body content field are specified as the target fields for keyword matching, and the search keywords entered by the user are used as the text expression to be matched.

[0048] The semantics of the combined search statement are: to find all records in the full-text index record set that contain the search keywords in the document title field or the document body content field.

[0049] 4) Filtering process for deleting marker fields When constructing the combined search statement, a filtering constraint is attached to the deletion marker field. This filtering constraint specifies that a record is included in the search scope only if the value of the deletion marker field in a full-text index record indicates that the document is not in a logical deletion state. By attaching this filtering constraint, full-text index records corresponding to documents marked as deleted are excluded from the search scope, thereby ensuring that the search results do not include documents for which the user has performed a logical deletion operation.

[0050] 5) Search execution and word segmentation matching The backend service program submits the constructed joint search statement and additional filtering constraints to the full-text search engine. Upon receiving the joint search statement, the full-text search engine performs word segmentation on the text content stored in the document title and document body content fields according to its internally configured word segmentation rules, and then performs a matching operation between the word segmentation results and the search keywords. For each document that simultaneously meets the keyword matching conditions and the deletion tag filtering conditions, the full-text search engine calculates a matching score, which represents the degree of relevance between the document content and the search keywords.

[0051] 6) Sorting process and generation of document identifier set After completing the matching operation, the full-text search engine sorts the result set in descending order based on the matching score of each matched document, placing documents with higher relevance to the search keywords at the beginning of the result sequence. After sorting, the engine extracts the values ​​of the unique document identifier fields from each record in the sorted result set, compiling them into a document identifier set for the matched documents. Each document identifier in this set uniquely corresponds to a document that meets the search criteria and has not been logically deleted.

[0052] S6. Link business metadata and generate search results After obtaining the document identifier set of the matched documents in step S5, step S6 is executed to integrate the search results with the business description information and generate the final return data that can be presented to the user. The specific implementation process of step S6 is as follows: 1) Retrospective lookup of business metadata The backend service program uses the document identifier set output in step S5 as input parameters to initiate a data query request to the business database. The query condition for the query request is that the value of the primary key identifier field of the document record in the business database belongs to the range listed in the document identifier set. The fields specified in the query request to be returned include business metadata items related to the display of document search results. These business metadata items at least include the document title, file size, update time, and file path.

[0053] 2) Assembly of business metadata and search results In response to the data query request, the business database returns business metadata records corresponding to each document identifier in the document identifier set. After receiving these business metadata records, the backend service program associates and assembles each record with the corresponding search-related information generated during the retrieval process in step S5. The search-related information may include the document's matching score or the keyword summary information in this retrieval. The output of the assembly operation is a structured document result entry, which simultaneously contains business metadata describing document management attributes and search feature data characterizing the relevance of the document to the search keywords.

[0054] 3) Generation of document result list For each document identifier in the document identifier set, an association assembly operation is performed to obtain a set of document result entries. These document result entries are then organized according to a preset sorting rule to form a document result list. The sorting rule can be determined based on the matching score ranking determined in step S5, or it can be based on a secondary sorting using the update time field in the business metadata.

[0055] 4) Returning search results The backend service program encapsulates the assembled list of document results into a search response data body and returns it to the client that initiated the search request. After receiving the search response data body, the client can visualize the search results based on the business metadata information contained therein, such as document title, file size, update time, and file path. The client can also initiate further access or download operations for the encrypted source document based on the file path information.

[0056] S7. Perform document change operations and index synchronization.

[0057] During the operation of the business system, when a change operation is performed on the document data in the business database, the system triggers index synchronization processing associated with the change operation to ensure that the index data stored in the full-text search engine and the document data in the business database maintain a correspondence in terms of content representation and deletion status marking.

[0058] The processing involved in step S7 specifically includes the following three aspects: index synchronization actions for different document change operation types, the triggering method for index synchronization processing, and consistency repair methods.

[0059] At the first level, the correspondence between index synchronization actions and document change operation types. When a document is added in the business system, an index addition process is triggered. This process includes: retrieving the business metadata and the extracted text content of the new document; constructing a full-text index record corresponding to the new document based on the metadata and text content; and writing this full-text index record to the index storage area managed by the full-text search engine. Once the index addition process is complete, the text content of the new document is included in the searchable scope of the full-text search engine.

[0060] When a document is modified in the business system, an index update process is triggered. This process includes: locating the existing full-text index record corresponding to the modified document in the full-text search engine based on its unique document identifier; and replacing or correcting the corresponding field values ​​in the existing full-text index record based on the modified document content and business metadata. The updated fields include at least the document content field and the update time field. After the index update process is complete, the index content of the document in the full-text search engine is consistent with the current content of the document in the business database.

[0061] When a document is logically deleted in the business system, an index deletion mark update process is triggered. This process includes: locating the existing full-text index record corresponding to the logically deleted document in the full-text search engine based on its unique document identifier; and modifying the value of the deletion mark field in that existing full-text index record to a preset value indicating a deleted status. After the index deletion mark update process is completed, the index record for that document will be excluded from the search results during subsequent full-text searches due to the filtering constraints of the deletion mark field.

[0062] When a document is physically deleted in the business system, an index deletion process is triggered. This process includes: locating the existing full-text index record corresponding to the physically deleted document in the full-text search engine based on its unique document identifier; and removing the existing full-text index record from the full-text search engine's index storage area. After the index deletion process is complete, the index data corresponding to the document no longer occupies storage space in the full-text search engine.

[0063] The second level is the triggering method for index synchronization processing. The index synchronization process for document change operations described above can be triggered by one or a combination of the following two methods: The first triggering method is a synchronization method directly triggered by the business interface. In this method, when a business system performs document addition, modification, logical deletion, or physical deletion operations through its provided business interface, the business interface directly calls the index synchronization processing logic simultaneously with or after processing business database write or update transactions to complete the corresponding index addition, index update, index deletion mark update, or index deletion actions. This method ensures that the index synchronization action and the business operation are within the same call chain, providing good real-time performance.

[0064] The second triggering method is a synchronization method based on database change monitoring. In this method, the system deploys a database change monitoring component that listens for change events in document-related tables within the business database. When the monitoring component captures an insert, update, or delete event for a document record, it parses the unique document identifier and change type involved in the change event and triggers the corresponding index synchronization processing logic based on the change type. This method decouples index synchronization processing from the business interface logic, allowing the business interface to focus only on the business database operations themselves.

[0065] The third level, consistency repair methods To address situations where index synchronization processing may fail due to network fluctuations, service anomalies, or malfunctions in the synchronization process itself, the solution outlined in step S7 also provides consistency repair methods. These consistency repair methods include retrying failed tasks and periodic verification repair.

[0066] The retry mechanism for failed tasks refers to the system recording relevant information about the failed task when an index synchronization task fails, and then re-initiating the index synchronization task for the same document and the same change type after a preset delay. This retry process can be repeated until synchronization is successful.

[0067] The scheduled verification and repair method refers to the system using a scheduled verification task to comprehensively compare the consistency status of document data in the business database with the index data in the full-text search engine, and to perform corresponding repair operations for any missing indexes, redundant indexes, or inconsistencies in deletion status found during the comparison process.

[0068] S8. Perform index consistency verification and repair. In addition to the incremental index synchronization mechanism described in step S7, step S8 provides the capability for periodic verification and automatic repair of the consistency status between business document data and full-text index data. The specific implementation of step S8 includes the following two levels of processing mechanisms: The first level is the retry mechanism for index synchronization tasks. If an index synchronization task fails due to network interruption, temporary service unavailability, or resource contention during the index synchronization process in step S7, a retry process for the failed task will be triggered.

[0069] The retry process involves recording the failure status and related context information of the synchronization task, and then re-initiating the same index synchronization operation as the failed task after a preset time interval. This retry operation can be configured to be repeated multiple times until the index synchronization task succeeds or the preset retry limit is reached. This retry mechanism reduces the risk of index synchronization interruptions due to occasional failures and improves the reliability of the incremental synchronization process.

[0070] The second level involves consistency verification and repair processing based on timed checks. The system is configured with a timed verification task, which automatically starts and executes according to a preset time period. A complete timed verification process includes the following steps: First, a data comparison operation is performed. The verification task retrieves the datasets to be compared from both the business database and the full-text search engine. The baseline dataset retrieved from the business database contains the document unique identifier for each document and its current deletion status value. The comparison dataset retrieved from the full-text search engine contains the document unique identifier field value and the deletion flag field value for each full-text index record.

[0071] Secondly, inconsistencies are identified based on the data comparison results. The baseline dataset and the comparison dataset are compared and linked according to document unique identifiers to identify and record the following three inconsistencies: The first type of inconsistency is a missing index. A missing index means that a document record with a unique document identifier exists in the business database, but the full-text search engine does not have a corresponding full-text index record.

[0072] The second type of inconsistency is redundant indexes. A redundant index refers to a situation where the full-text search engine has a full-text index record corresponding to a unique document identifier, but the business database does not have a document record corresponding to that unique document identifier.

[0073] The third type of inconsistency is the inconsistency in deletion status. Inconsistent deletion status means that for the same unique document identifier that exists in both the business database and the full-text search engine, the deletion status value of the document in the business database does not match the deletion flag field value of the record in the full-text search engine.

[0074] Finally, the inconsistency correction process is executed. For the three inconsistencies identified above, the verification task takes corresponding corrective actions: In the case of missing indexes, a corresponding full-text index record is constructed based on the current metadata of the document in the business database and the main text content that has been re-acquired, decrypted and extracted, and then written into the full-text search engine.

[0075] To address redundant indexes, the redundant full-text index records corresponding to the unique identifier of the document in the full-text search engine will be deleted.

[0076] In cases of inconsistent deletion status, the value of the deletion flag field of the corresponding record in the full-text search engine will be updated to match the value of the deletion status of the document in the business database.

[0077] In some embodiments, the encrypted document storage system, decryption service, document parsing module, full-text search engine, and business database are deployed in the same internal secure network environment to reduce the security risks associated with plaintext data transmission over external networks.

[0078] Example The vulnerability knowledge base full-text search method in this embodiment is implemented based on an overall architecture, see [link to documentation]. Figure 1 The technical architecture upon which this method is based includes the following components: The business database is used to store metadata related to document management, including document identifier, document title, file path, update time, file size, and deletion status.

[0079] An encrypted document storage system is used to store the document entity data of vulnerability knowledge documents in encrypted form.

[0080] The decryption service is used to perform temporary decryption processing on encrypted document data obtained from the self-encrypted document storage system to obtain plaintext data that can be extracted from the text.

[0081] The document parsing module is used to perform format recognition and text extraction operations on the temporarily decrypted plaintext data.

[0082] The full-text search engine is used to build full-text index records based on document metadata and text content, and to perform keyword matching and hit calculation in response to search requests.

[0083] Encrypted storage volumes are used to hold the index data of the full-text search engine, ensuring that the index data is stored in encrypted form at the persistent storage level.

[0084] The backend service establishes connections with the frontend search interface, business database, decryption service, and full-text search engine, respectively, and is used to receive search requests, schedule decryption and parsing processing, construct search statements, and assemble and return search results.

[0085] The front-end search interface is used to receive search keywords entered by users and to visualize the search results.

[0086] The connections between the above components are as follows: the business database is connected to the backend service, providing a metadata query channel for the backend service; the encrypted document storage system is connected to the decryption service, providing encrypted document data to the decryption service; the decryption service is connected to the document parsing module, whereby the decryption service transmits temporary plaintext data to the document parsing module for text extraction; the backend service is connected to the full-text search engine, used to issue index building instructions and search requests; the full-text search engine is connected to the encrypted storage volume, writing index data to the encrypted storage volume or reading index data from the encrypted storage volume; and the frontend search interface is connected to the backend service, used to submit search requests and receive search result data.

[0087] 1) Index initialization and construction process In the scenario of initial system deployment or first-time activation of full-text search function, the index initialization and construction process is executed, which includes the following processing stages.

[0088] The first phase involves batch reading of metadata. The backend service initiates a query request to the business database to retrieve the set of metadata records corresponding to all currently stored vulnerability knowledge documents. Each metadata record contains at least the document identifier, document title, file path, update time, and deletion status.

[0089] The second phase involves the acquisition and temporary decryption of encrypted documents. For each document in the acquired metadata record set, the backend service reads the corresponding encrypted document data from the encrypted document storage system based on its file path and transmits the encrypted document data to the decryption service. After receiving the encrypted document data, the decryption service requests the decryption key information corresponding to the document from the key management service and uses the decryption key information to perform decryption operations on the encrypted document data, generating temporary plaintext data. This temporary plaintext data is stored only in the memory area of ​​the computing device performing the decryption operation and is not written to the local disk or any persistent storage medium.

[0090] The third stage is text extraction. The document parsing module identifies the format of the temporary plaintext data stored in memory. Supported document formats include plain text, portable document formats, and document formats generated by word processing software. After determining the document format, the document parsing module extracts the main text content of the document. After the main text content extraction operation is completed, the temporary plaintext data in memory is actively released or overwritten and destroyed.

[0091] The fourth stage involves the construction and writing of full-text index records. For each document, a unique full-text index record is constructed based on its metadata, including the document identifier, document title, file path, update time, deletion status, and the text content extracted in the third stage. The fields of this full-text index record are determined as follows: the document identifier field uniquely identifies the document and its associated business database; the document title field is included in the search matching scope by the full-text search engine; the text content field is included in the search matching scope by the full-text search engine; the file path field records the storage location of the encrypted document; the update time field serves as the sorting basis for search results; and the deletion flag field filters logically deleted document records during retrieval.

[0092] The completed full-text index records are written to the index storage area managed by the full-text search engine. The full-text search engine performs word segmentation on the document title field and the body content field and builds an inverted index structure. The full-text search engine's index data is stored in an encrypted storage volume and saved in ciphertext form at the disk level; when a search or read operation is performed, the file system layer where the encrypted storage volume resides transparently decrypts the ciphertext index data and delivers the plaintext index data to the full-text search engine for use.

[0093] 2) Full-text search execution process When a user enters search keywords and initiates a search request through the front-end search interface, the following processing flow is executed.

[0094] First, the backend service receives the search request, obtains the user's identity information, and verifies whether the user has document search permissions according to preset permission rules. If the verification fails, the search process is terminated and a permission denied message is returned.

[0095] After successful verification, the backend service constructs a combined search statement for the full-text search engine. This combined search statement specifies the document title and body content fields as the fields to be matched, and uses the user-input search keywords as the matching expression. Simultaneously, the combined search statement adds a filtering constraint on the deletion flag field, limiting the scope of the indexed record to only those records whose deletion flag value indicates that the document is not in a logically deleted state.

[0096] After receiving the combined search query, the full-text search engine performs word segmentation on the text content stored in the document title and body content fields, and matches it with the search keywords. For each document that matches the search criteria and satisfies the deletion flag filtering constraints, a matching score is calculated. Based on the matching score, the full-text search engine sorts the matching documents in descending order, extracts the document identifier field value from each record in the sorted result set, forms a document identifier set, and returns it to the backend service.

[0097] After obtaining the document identifier set, the backend service uses these document identifiers as the query basis to request the corresponding document's metadata information from the business database. The obtained metadata information includes at least the document title, file size, update time, and file path. The backend service then associates and assembles the matching ranking information returned by the full-text search engine with the metadata information returned by the business database to generate a list of document results that can be displayed on the frontend, and returns this list of document results to the frontend search interface.

[0098] 3) Index synchronization processing in document change scenarios During the operation of the business system, when operations such as adding, modifying, logically deleting, or physically deleting document data occur, the corresponding index synchronization process is triggered to maintain the consistency between the document data in the business database and the index data in the full-text search engine.

[0099] A new document entry triggers an index creation process. The backend service retrieves the metadata of the new document and the text content obtained through temporary decryption and parsing, constructs a new full-text index record, and writes this record to the full-text search engine.

[0100] Document modification operations trigger index update processing. The backend service locates the corresponding existing index record in the full-text search engine based on the document identifier of the modified document, and updates the text content field and update time field of the index record according to the modified document text content and update time.

[0101] A document logical deletion operation triggers an index deletion flag update. The backend service locates the corresponding existing index record in the full-text search engine based on the document identifier of the logically deleted document, and updates the value of the deletion flag field in that index record to indicate that it has been deleted.

[0102] The physical deletion of a document triggers index deletion processing. The backend service locates the corresponding existing index record in the full-text search engine based on the document identifier of the physically deleted document, and removes the index record from the full-text search engine's index storage area.

[0103] The above index synchronization processing can be implemented by one or a combination of two triggering methods: one is to trigger directly based on the business interface, that is, to synchronously call the index synchronization processing logic during the transaction process of document change operation in the business interface; the other is to trigger based on database change monitoring, that is, to trigger the corresponding index synchronization processing after monitoring the change events of document-related data tables in the business database, parsing the change type and the affected document identifier.

[0104] 4) Index consistency guarantee mechanism To address potential execution failures or data discrepancies during index synchronization, this method establishes an index consistency guarantee mechanism, which includes two aspects: failure retry handling and periodic verification and repair handling.

[0105] At the failure retry handling level, when an index synchronization task fails due to network or service anomalies, the system records the relevant context information of the failed task and re-initiates the same type of index synchronization task after a preset delay time. The retry operation can be repeated until the task succeeds or the preset retry limit is reached.

[0106] At the scheduled verification and repair level, the system is configured to perform verification tasks periodically. These tasks retrieve a baseline dataset of current document identifiers and their deletion status from the business database, and a comparison dataset of document identifiers and deletion marker fields from the full-text search engine within the current indexed records. By comparing the baseline and comparison datasets by document identifier, the system identifies three inconsistencies and performs corresponding repair actions: If a document identifier exists in the business database but there is no corresponding index record in the full-text search engine, it is determined to be a missing index. An index record is constructed based on the current metadata of the document and the re-extracted text content and written to the full-text search engine. If a document identifier exists in the full-text search engine but the corresponding document record does not exist in the business database, it is determined to be a redundant index, and the redundant index record is deleted from the full-text search engine. If the deletion status of the same document identifier in the business database is inconsistent with the deletion status represented by the value of the deletion marker field in the full-text search engine, then the deletion marker field value of the corresponding record in the full-text search engine will be corrected according to the deletion status in the business database.

[0107] Although the embodiments of this application have been described above in conjunction with the accompanying drawings, this application is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of this application, and these are all within the scope of protection of this application.

Claims

1. A full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction, characterized in that, Includes the following steps: S1. Read the business metadata of the document from the business database, and obtain the corresponding encrypted document data from the encrypted document storage system according to the file path in the business metadata; S2. Call the decryption service to decrypt the encrypted document data and generate temporary plaintext data that is only temporarily stored in memory; call the document parsing module to extract the text from the temporary plaintext data to obtain the main text content of the document, and release or destroy the temporary plaintext data after the extraction is completed. S3. Based on the business metadata and the main text content, establish a corresponding full-text index record for the document and write the full-text index record into the full-text search engine; wherein, the full-text index record includes at least a document unique identifier field, a document title field, a document main text content field, and a deletion flag field; S4. Store the index data of the full-text search engine in an encrypted storage volume so that the index data is saved in encrypted form at the disk layer; S5. Receive keywords input by the user, construct a joint search statement for the document title field and the document body content field, filter deleted documents by combining the deletion mark field, and perform a search in the full-text search engine to obtain a set of document identifiers of the hit documents. S6. Based on the document identifier set, query the business database to obtain the business metadata of the hit document, and then combine the business metadata with the search results and return it. S7. When a document is added, modified, logically deleted, or physically deleted, the corresponding index synchronization process is triggered to maintain the consistency between business document data and full-text index data.

2. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security indexing as described in claim 1, characterized in that, The business metadata in step S1 includes the document's unique identifier, document title, file path, update time, and deletion status.

3. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction according to claim 1, characterized in that, The document parsing module in step S2 is adapted to extract text from various document formats, including txt, pdf, doc, and docx.

4. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction according to claim 1, characterized in that, The full-text index record in step S3 also includes a file path field and an update time field; wherein, the document title field and the document body content field are used as retrieval fields, and the update time field is used as a sorting field.

5. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction according to claim 1, characterized in that, Step S5 also includes: verifying the user's search permissions before performing the search, and constructing the combined search statement after the verification is passed.

6. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security indexing as described in claim 1, characterized in that, The index synchronization process in step S7 includes: When a document is added, a new full-text index record is created; When a document modification operation occurs, update the text content and update time in the corresponding full-text index record; When a document is logically deleted, the deletion flag field in the corresponding full-text index record will be updated to indicate that it has been deleted. When a document is physically deleted, the corresponding full-text index record is deleted.

7. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction according to claim 1, characterized in that, The index synchronization process in step S7 is triggered either directly through the business interface or through database change monitoring.

8. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction according to claim 1, characterized in that, The method also includes the step of performing index consistency verification and repair: by using a timed verification method, the document data in the business database is compared with the index data in the full-text search engine, and the situation of missing indexes, redundant indexes or inconsistent deletion status is repaired.

9. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction according to claim 8, characterized in that, When index synchronization fails, the synchronization task is retried through a retry mechanism.

10. The full-text retrieval method for a vulnerability knowledge base based on encrypted document parsing and security index construction according to claim 1, characterized in that, The encrypted document storage system, decryption service, document parsing module, full-text search engine, and business database are deployed in the same internal secure network environment.

Images