A cloud database-based digital archive information storage method and system

Abstract

The application relates to the technical field of cloud databases, and discloses a digital archive information storage method and system based on a cloud database, which comprises the following steps: receiving a query request containing multiple query conditions, decomposing the query request into multiple independent query parts, distributing the query parts to corresponding processing units according to the data characteristics of the query parts, deploying a processing module on each data shard, maintaining a semantic information index related to local archives, preliminarily screening the local archives according to the received query parts and the semantic information index, and obtaining simplified archive identification information. Finally, archive identification information from multiple processing modules is gathered, and archive content is obtained according to the gathered archive identification information. Through the technical scheme, the application can effectively deal with the complex correlation query and frequent update challenges of digital archives in a distributed cloud database environment.

Description

Technical Field

[0001] This invention relates to the field of cloud database technology, and more specifically, to a method and system for storing digital archival information based on a cloud database. Background Technology

[0002] In modern information management, especially for organizations that need to process massive amounts of historical or administrative records, the storage and management of digital archives presents enormous challenges. Traditional storage and retrieval systems often succumb to the sheer volume of digital data and the ever-changing needs of users. Managing digital archives, particularly in cloud database environments, presents significant challenges when data is distributed across multiple storage locations. Ensuring rapid information retrieval, handling frequent updates, and maintaining data consistency within a distributed system become critical issues.

[0003] For example, a system designed by an organization to manage massive amounts of digitized archival information is built on a distributed cloud database framework. To efficiently handle the ingestion and initial retrieval of archives, the system employs a data sharding rule based on the time range of archive creation. This design strategy initially demonstrated significant advantages, ensuring a uniform distribution of newly ingested archival data and simplifying historical data queries by time range. However, as digitized archival data continues to accumulate, users' needs for utilizing archival information have also undergone profound changes. Traditional, single-time-range retrieval models can no longer meet increasingly complex analytical needs; users now prefer to perform complex content-related queries across multiple archive types and containing specific keywords. This query model requires the system to understand and associate the deep semantic relationships between different archival entities, rather than simply filtering based on descriptive attributes.

[0004] This new query pattern poses a significant challenge to the distributed query optimizer of existing systems. The optimizer's initial design and optimization strategy were built around single-shard or neighboring-shard queries based on time ranges. It excels at handling queries with high data locality, efficiently retrieving and aggregating data within a few shards. However, when faced with complex relational queries across multiple file types and keywords, the related data may be scattered across dozens or even hundreds of non-adjacent shards. This forces the query optimizer to initiate numerous cross-shard data transfers and result aggregation operations. This not only results in significant network input / output overhead but also increases the computational burden of data preparation, transformation, and complex aggregation and join operations at the central node, significantly increasing query response time and severely impacting user experience.

[0005] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0006] This invention provides a method for storing digital archival information based on cloud databases, aiming to solve the technical problems of low efficiency of traditional storage and retrieval systems when dealing with massive amounts of digital archival data in a distributed cloud database environment, as well as the large network I / O overhead and heavy computational burden caused by cross-shard queries, which affect user experience.

[0007] The technical solution of this application is as follows: Firstly, this application discloses a method for storing digital archival information based on a cloud database, used to process updates of archival description information and complex relational queries in a distributed cloud database environment where archival data is fragmented according to creation time range, including: It receives query requests containing multiple query conditions and breaks them down into multiple independent query parts; Based on the data characteristics of each query part, the query parts are distributed to the corresponding processing units; On each data shard, a processing module is deployed. The processing module maintains a semantic information index related to local archives. Based on the received query and the semantic information index, the local archives are initially filtered to obtain concise archive identification information. It aggregates file identification information from multiple processing modules and retrieves file content based on the aggregated file identification information.

[0008] This technical solution effectively addresses the challenges of complex relational queries and frequent updates of digital archives in a distributed cloud database environment. By decomposing and distributing queries to corresponding processing units and performing preliminary screening on each data shard, it significantly reduces the burden of cross-shard data transmission and central node aggregation, thereby improving query efficiency and system response speed, and solving the problem of low efficiency in complex queries in existing technologies.

[0009] Furthermore, based on the above, this application also proposes to deploy a processing module on each data shard. This processing module maintains a semantic information index related to local archives and performs preliminary screening of local archives based on the received query portion and the local semantic information index to obtain concise archive identification information, including: Within the processing module, an in-memory index is maintained based on fine-grained tags and link information; Deploy a local graph relation database based on semantic association information; Maintain an inverted text index based on keyword and tag information; Set up a routing and transformation layer to coordinate the updates and queries of the in-memory index, the local graph relational database, and the inverted text index; On each data shard, semantic metadata and its index are stored in an independent metadata hotspot. When the query requires the original content of the file, the content on-demand activation mechanism is activated to retrieve the file content from the low-cost storage layer.

[0010] This technical solution integrates an in-memory index, a local graph relational database, and an inverted text index within the processing module, and sets up a routing and transformation layer for coordination. This enables refined management and efficient retrieval of archival semantic information. At the same time, by combining metadata independent hot zone storage and content on-demand activation mechanism, storage costs and retrieval efficiency are optimized, overcoming the limitations of a single index structure in handling complex semantic queries.

[0011] More specifically, in some implementations, within the processing module, a memory index is maintained based on fine-grained tags and linking information, including: The memory index is divided into multiple logical partitions, and each logical partition is configured with independent memory resources. When an update request for fine-grained tags or link information is received, the update request is routed to the corresponding logical partition for processing based on the file element identifier of the update request. Within each logical partition, concurrent updates are handled using lock-free data structures or optimistic locking mechanisms to reduce lock contention. Fine-grained tag and link information is periodically written incrementally to persistent storage media in the metadata independent hot zone; During system startup or fault recovery, fine-grained tag and link information is loaded from persistent storage media into the memory index.

[0012] This technical solution significantly improves the concurrent processing capability and data consistency of the memory index by dividing the memory index into logical partitions and using lock-free or optimistic locking mechanisms to handle concurrent updates. At the same time, combined with incremental persistence and fault recovery mechanisms, it ensures the reliability and availability of fine-grained tag and link information, effectively solving the performance bottleneck in high-concurrency update scenarios.

[0013] Preferably, based on the above, this application further proposes to deploy a local graph relation database according to semantic association information, including: The graph structure of the local graph relational database is divided into a core semantic layer and an extended semantic layer; The core semantic layer stores frequently accessed and strongly related semantic nodes and edges, and adopts a memory-optimized graph storage structure. In the extended semantic layer, low-frequency access and weakly related semantic nodes and edges are stored, and a disk-optimized graph storage structure is adopted. When a semantic association query request is received, graph traversal and query are performed in the core semantic layer to obtain preliminary query results. When the initial query results are incomplete, supplementary queries are performed in the extended semantic layer according to the extended requirements of the semantic association query request to obtain supplementary query results. The initial query results and supplementary query results are then integrated to obtain the final query results.

[0014] This technical solution divides the local graph relational database into a core semantic layer and an extended semantic layer, and adopts memory-optimized and disk-optimized storage structures respectively. This enables differentiated management of semantic information with different access frequencies and association strengths, thereby effectively controlling storage costs while ensuring high-frequency query response speed. Furthermore, it can flexibly respond to semantic association query needs of different complexities through a layered query mechanism.

[0015] Building upon the above, this application further proposes maintaining an inverted text index based on keyword and tag information, including: The inverted text index is divided into a core index layer and an extended index layer; The core index layer stores frequently accessed keyword and tag information and uses a memory-optimized data structure; The extended index layer stores frequently accessed keyword and tag information and uses a disk-optimized data structure; When an update request for keyword or tag information is received, the update request is routed to the core index layer or the extended index layer for updating, depending on the activity level of the keyword or tag in the update request. When a query request is received, a query is performed in the core index layer to obtain preliminary query results. If the preliminary query results are incomplete, a supplementary query is performed in the extended index layer to obtain supplementary query results. The preliminary query results and supplementary query results are then integrated to obtain the final query results. Content-sensitive inverted list compression technology compresses the inverted list in the inverted text index.

[0016] This technical solution divides the inverted text index into a core index layer and an extended index layer, and stores and updates them in layers according to the activity of keywords and tags. This effectively balances query performance and storage costs. At the same time, it introduces content-sensitive inverted list compression technology to further optimize storage efficiency and solve the storage and query efficiency problems faced by traditional inverted indexes when dealing with massive amounts of data.

[0017] Building upon this, this application also proposes a content-sensitive inverted list compression technique to compress the inverted list in the inverted text index, including: Obtain sensitive information about the contents of the archives; Based on sensitivity information and preset compression algorithm performance parameters, a compression algorithm is dynamically selected to compress the inverted list in the inverted text index; When the sensitivity information of the archive content changes, the sensitivity level of the archive content is reassessed, and the compression algorithm is dynamically adjusted according to the new sensitivity level and the preset compression algorithm performance parameters. When compressing the inverted list in the inverted text index, sensitivity information of the archive content is embedded into the compressed data structure; Before compressing the inverted list in the inverted text index, the terms in the inverted list are divided into sensitive terms and non-sensitive terms according to the sensitivity level of the archive content, and then stored separately. When a query request is received that crosses fragments and sensitivity levels, the inverted list of different compression strategies is identified before decompression based on the embedded sensitivity information. When merging and deduplicating inverted lists using different compression strategies, a unified intermediate data format is used for conversion, and merging and deduplication operations are performed after conversion.

[0018] This technical solution, by introducing a content-sensitive inverted list compression technology, achieves differentiated compression and storage of archive content with different sensitivities. This not only improves storage efficiency but also enhances data security and query flexibility by embedding sensitivity information and dynamically adjusting the compression algorithm. It effectively solves the problem that traditional compression technologies cannot simultaneously achieve storage efficiency, security, and query efficiency.

[0019] Building upon the above, this application further proposes that when merging and deduplicating inverted lists using different compression strategies, a unified intermediate data format is used for conversion, followed by merging and deduplication operations, including: A block processing mechanism is used to divide the data stream of the inverted list into multiple data blocks; Each data block is decompressed and format converted independently to obtain the conversion result; Store the transformation results based on the local cache; The conversion results are transferred to an intermediate storage area using an asynchronous write method. By parallelizing the processing, multiple data blocks in the intermediate storage area are distributed to different computing threads or processes; In each computing thread or process, local deduplication is performed on the received data blocks, and the processed terms and file identifiers are recorded based on a hash-based data structure; The results of local deduplication are summarized and then deduplicated to obtain the final merged deduplication result.

[0020] This technical solution significantly improves the efficiency of merging and deduplication across shards and across sensitivity levels of inverted lists by using block processing, parallel decompression and format conversion, as well as local deduplication and final summary deduplication mechanisms. It effectively solves the performance bottleneck of data deduplication and integration in a large-scale distributed environment and ensures the accuracy and real-time performance of query results.

[0021] Building upon the above, this application further proposes that, within each computation thread or process, the received data blocks be locally deduplicated, and that processed terms and file identifiers be recorded based on a hash-based data structure, including: In each computing thread or process, when performing local deduplication on the received data block, the data is mapped to one of multiple hash functions based on the characteristics of the term or file identifier to obtain the first mapping result; When a hash collision occurs in the first mapping result, the conflicting term or file identifier is attempted to be mapped to another preset hash function to obtain a second mapping result; If the second mapping result still conflicts, the conflicting term or file identifier is stored in an overflow area. When recording processed terms and file identifiers, they are stored in different hash table partitions based on the activity level of the terms or file identifiers. For highly active terms or file identifiers, allocate separate hash table partitions and use a memory-optimized hash structure.

[0022] This technical solution significantly improves the efficiency and accuracy of local deduplication operations by employing multiple hash function mapping and overflow region to handle hash collisions, as well as partitioning the hash table according to activity levels. In particular, the use of a memory-optimized hash structure for highly active data further enhances processing performance and effectively solves the problems of hash collisions and performance bottlenecks in large-scale data deduplication.

[0023] Building upon the above, this application further proposes allocating independent hash table partitions for highly active terms or file identifiers, and employing a memory-optimized hash structure, including: In each computation thread or process, a separate hash table partition is assigned to highly active terms or file identifiers; Receive new terms or file identifiers and evaluate the activity of terms or file identifiers; Based on the activity level and a preset activity threshold, determine whether a term or file identifier belongs to high-activity data; Highly active data is mapped to corresponding hash table partitions using a hash function; Within the hash table partition, a memory-optimized hash structure based on Cuckoo hashing is used, and a certain proportion of spare storage space is reserved during the initialization of the hash structure. When a hash table partition approaches its capacity limit, a dynamic expansion process is initiated. This process is accomplished by allocating new memory blocks in the background and gradually migrating data.

[0024] This technical solution significantly improves the processing efficiency and query performance of high-activity data by allocating independent hash table partitions for high-activity terms or file identifiers and adopting a memory-optimized hash structure based on Cuckoo hashing. At the same time, combined with a dynamic expansion mechanism, it ensures the stability and scalability of the system under high load, effectively solving the performance bottleneck and capacity limitation problems that may occur in traditional hash structures for high-activity data.

[0025] Secondly, this application also discloses a digital archival information storage system based on a cloud database, used to process archival description information updates and complex relational queries in a distributed cloud database environment where archival data is fragmented according to creation time range, including: The input end is used to receive query requests containing multiple query conditions and decompose the query request into multiple independent query parts; according to the data characteristics of each query part, the query part is distributed to the corresponding processing unit; The processing end is used to deploy a processing module on each data shard. The processing module maintains a semantic information index related to local archives and performs preliminary screening of local archives based on the received query part and semantic information index to obtain concise archive identification information. The output end is used to aggregate file identification information from multiple processing modules and obtain file content based on the aggregated file identification information.

[0026] This application provides a system-level solution through the collaborative work of the input end, processing end and output end, which realizes the effective decomposition, distributed processing and result aggregation of complex query requests, significantly improves the efficiency and accuracy of processing digital archive information updates and complex relational queries in a distributed cloud database environment, and solves the problem of insufficient performance of existing systems when facing massive data and complex queries.

[0027] The beneficial effects of this invention are as follows: This application discloses a method and system for storing digital archival information based on a cloud database. It aims to solve the technical problems in the prior art where, in a distributed cloud database environment, traditional storage and retrieval systems are inefficient when dealing with frequent updates and complex relational queries on massive amounts of digital archival data, and where cross-shard queries result in high network I / O overhead and heavy computational burden, thus affecting user experience.

[0028] This application receives query requests containing multiple query conditions, decomposes them into multiple independent query parts, and distributes them to corresponding processing units based on the data characteristics of each query part. On each data shard, a processing module is deployed to maintain a semantic information index related to local archives, and performs preliminary screening of local archives based on the received query parts and the semantic information index to obtain concise archive identification information. Finally, the archive identification information from multiple processing modules is aggregated, and the archive content is obtained based on the aggregated archive identification information.

[0029] Through the above technical solution, this application can effectively address the challenges of complex relational queries and frequent updates of digitized archives in a distributed cloud database environment. Specifically, by decomposing complex query requests and distributing them to various data shards for local processing, the amount of data transmission across shards and the aggregation computation burden on the central node are significantly reduced. The processing module deployed on each data shard utilizes local semantic information indexes to perform preliminary screening of archives, enabling rapid and accurate location of relevant archives, thereby significantly improving query efficiency and system response speed. This distributed, localized processing strategy effectively overcomes the performance bottleneck of the query optimizer caused by data being scattered across non-adjacent shards in existing technologies, reduces network I / O overhead and the computational burden of data preparation and transformation, thus significantly improving the user experience. Therefore, the method of this application has significant and superior technical effects in handling complex relational queries and frequent updates of massive amounts of digitized archives. Attached Figure Description

[0030] Figure 1 This is a flowchart of a method for storing digital archival information based on a cloud database, provided by an embodiment of the present invention. Figure 2 This is a flowchart of a method for obtaining simplified file identification information provided by an embodiment of the present invention; Figure 3 This is a flowchart of a method for maintaining a memory index provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a digital archive information storage system based on a cloud database provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Reference Figure 1 , Figure 1 This is a flowchart of a method for storing digital archival information based on a cloud database, provided by an embodiment of the present invention. It is used in a distributed cloud database environment where archival data is fragmented according to creation time ranges to process updates of archival description information and complex relational queries, including: S11, Receive a query request containing multiple query conditions, and decompose the query request into multiple independent query parts; S12, based on the data characteristics of each query part, distribute the query part to the corresponding processing unit; S13, on each data shard, a processing module is deployed. The processing module maintains a semantic information index related to local files and performs preliminary screening of the local files based on the received query portion and the semantic information index to obtain simplified file identification information.

[0034] S14, aggregate file identification information from multiple processing modules, and obtain file content based on the aggregated file identification information.

[0035] This application effectively solves the problem of low efficiency in complex relational queries in traditional systems by decomposing and distributing complex query requests to corresponding processing units, deploying processing modules on each data shard for preliminary screening, and finally aggregating concise file identification information to obtain file content. This significantly improves query response speed and user experience.

[0036] To better understand the technical solution proposed in this application, some key terms and implementation environment involved are first explained. The environment in which this application is based is a distributed cloud database environment, in which archival data is stored in shards according to the creation time range. This means that the archival data is logically or physically divided into multiple independent data blocks, each data block is called a "data shard", and may be stored on different servers or storage nodes. This sharding mechanism helps to improve data scalability and query parallelism.

[0037] "Archival description information" refers to metadata related to the content of an archive, such as the archive's title, author, creation date, keywords, and category tags. This information is typically used for archive retrieval and management. "Semantic information index" is a high-level index structure that includes not only keywords or tags but also semantic relationships between archives and between elements within an archive, such as entity relationships and concept hierarchies represented by a graph structure. This type of index supports more complex, semantically understanding-based queries, rather than simply text matching.

[0038] A "query request" refers to an instruction initiated by a user or system to retrieve archival information. It may contain multiple query conditions, such as time range, keywords, archival type, and semantic relationships. A "query component" refers to a subquery that can be processed independently, derived from breaking down a complex query request. A "processing unit" refers to a computational entity responsible for executing the data processing logic of a specific query component; it can be an independent server, a container instance, or a process.

[0039] The "processing module" is a software component deployed on each data shard. Its core function is to maintain a semantic information index related to local archives and to perform preliminary filtering of local archives based on the received query portion to generate concise archive identification information. "Archive identification information" refers to lightweight information that can uniquely identify an archive, such as an archive ID, archive path, or pointer to archive content. It does not contain the complete archive content and is designed to reduce the amount of data transmitted.

[0040] "Content on-demand activation mechanism" refers to a strategy in which the system retrieves the content of an archive from a low-cost storage layer (such as object storage or archive storage) only when the query results require the actual content of the archive, in order to optimize storage costs and retrieval efficiency.

[0041] The core of the digital archival information storage method based on cloud database proposed in this application lies in the intelligent processing and distributed execution of complex query requests.

[0042] First, when the system receives a query request containing multiple query conditions, such as a query that might include conditions like "during 2020 to 2022," "reports on artificial intelligence," and "documents related to a certain project," the system will decompose the query request into multiple independent query parts to efficiently handle this complexity. For example, "during 2020 to 2022" can be treated as a time range query, "reports on artificial intelligence" as a keyword query, and "documents related to a certain project" as a semantic association query. This decomposition can be accomplished by a syntax parser or a rule-based query optimizer. For instance, a query parser can identify logical operators (AND, OR, NOT) and different query predicates in the query string, thereby breaking down a complex SQL or NoSQL query into smaller, independently executable subquery units.

[0043] Secondly, based on the data characteristics of each query portion, the query portion is distributed to the corresponding processing unit. For example, a time range query portion might be distributed to the processing unit responsible for managing data shards for the corresponding time period; a keyword query portion might be distributed to all data shards that may contain relevant keywords; and a semantic association query portion might be distributed to the data shard that maintains a graph-related index. This distribution strategy can be based on the query optimizer's decisions, which consider factors such as the distribution of data shards, index types, and query locality. For example, a hash-based distributor can route queries to pre-configured processing units based on the hash values ​​of specific fields contained in the query portion.

[0044] Furthermore, a processing module is deployed on each data shard. This module maintains a semantic information index related to local archives and performs preliminary filtering of the local archives based on the received query portion and the semantic information index, obtaining concise archive identification information. For example, when a data shard receives a keyword query portion about "artificial intelligence," its processing module will use its local inverted index to quickly find archives containing "artificial intelligence" and return the identifiers of these archives, rather than the complete archive content. When a semantic relational query is received, the processing module will use its local graph relational database to perform graph traversal to find archive identifiers that match the query conditions. The purpose of this preliminary filtering is to minimize the amount of data that needs to be transmitted and processed at the data source end, thereby improving efficiency. For example, the processing module could be a lightweight service that listens for query portions from a central query coordinator and uses a local index structure (such as a B-tree index, hash index, or full-text index) to quickly filter the archive metadata stored on that shard.

[0045] Finally, the system aggregates file identification information from multiple processing modules and retrieves the file content based on this aggregated information. After the processing modules on all relevant data shards complete the initial screening and return file identification information, the central coordinator or aggregation service collects this information. Because this identification information is concise, transmission overhead is low. After aggregation, the system retrieves the complete file content from the storage layer (e.g., object storage or file system) on demand based on this file identification information. For example, if a user only needs to view the file's title and summary, the system may only return metadata; only if the user clicks on a file to view its full text will the system retrieve the complete file content based on the file identification information. This on-demand retrieval mechanism effectively reduces storage and network transmission costs. For example, an aggregation service can receive a list of file IDs from different processing modules, then deduplicate and sort these IDs, and finally initiate batch content retrieval requests to the underlying storage system based on these unique file IDs.

[0046] The digital archival information storage method based on cloud database proposed in this application can effectively solve the challenges faced by traditional distributed cloud databases in handling complex relational queries through the synergistic effect of the above steps.

[0047] refer to Figure 2 , Figure 2 This is a flowchart of a method for obtaining simplified file identification information provided in an embodiment of the present invention, including: S131, within the processing module, a memory index is maintained based on fine-grained tags and link information; S132, Deploy a local graph relation database based on semantic association information; S133, maintain an inverted text index based on keyword and tag information; S134, Set up a routing and transformation layer to coordinate the updating and querying of the memory index, the local graph relational database and the inverted text index; S135, on each data shard, semantic metadata and its index are stored in an independent metadata hotspot; S136, when the query requires the original content of the file, the content on-demand activation mechanism is activated to retrieve the file content from the low-cost storage layer.

[0048] Specifically, within the processing module, an in-memory index is maintained based on fine-grained tags and link information. This in-memory index is primarily used to store fine-grained tags and internal links within the archive that require fast access and frequent updates. Examples include archive category tags, version links, and reference relationships. This information typically has high access frequency and low data volume, making it suitable for storage in memory to achieve millisecond-level query responses.

[0049] Based on semantic association information, a local graph relational database is deployed. This database aims to efficiently store and query complex semantic relationships between archives, such as inheritance relationships, thematic associations, and interpersonal relationships. The graph structure facilitates multi-hop queries and path analysis, revealing deeper connections within the archive data.

[0050] In practical applications, an inverted text index is maintained based on keyword and tag information. This inverted text index is primarily used to support full-text search or fuzzy queries based on keywords and tags. For example, users can quickly locate files containing specific content by entering keywords, or filter by tags.

[0051] Furthermore, a routing and transformation layer is set up to coordinate the updates and queries of the in-memory index, the local graph relational database, and the inverted text index. This routing and transformation layer serves as the unified entry point for the processing modules, responsible for parsing incoming query requests, routing them to the most suitable underlying index for processing, and coordinating data synchronization and result integration between different indexes. Its purpose is to provide a unified and efficient interface, shielding the complexity of the underlying indexes.

[0052] In addition, semantic metadata and its index are stored in a dedicated metadata hotspot on each data shard. This dedicated metadata hotspot is an area specifically designed to store semantic metadata and its index, aiming to provide high-throughput and low-latency access capabilities, ensuring the persistence and fast retrieval of semantic information.

[0053] When a query requires the original content of a file, an on-demand content activation mechanism is activated, retrieving the file content from a low-cost storage layer. This means that the original content of a file is usually stored in a lower-cost storage medium (such as object storage), and is only loaded into memory or a faster storage layer on demand when the user explicitly requests to view the original content, thereby effectively reducing storage costs and improving the overall efficiency of the system.

[0054] The solution proposed in this application effectively solves the performance bottleneck of traditional single indexes when dealing with complex archive information updates and diverse related queries by refining the semantic information index into multiple specialized index structures.

[0055] In some embodiments described above, the processing module maintains an in-memory index to store fine-grained tags and link information. However, during implementation, when the amount of archival data is large and update operations are frequent, a single in-memory index may face problems such as concurrent access conflicts, low update efficiency, and insufficient data persistence, thereby affecting the overall performance and data reliability of the system. If these problems are not addressed, the scalability and stability of the in-memory index will be limited, making it difficult to meet the needs of large-scale digitized archival information storage. Therefore, this application further proposes a method to optimize in-memory index maintenance, effectively improving the performance and reliability of the in-memory index through refined management and concurrency control mechanisms.

[0056] In this regard, refer to Figure 3 , Figure 3 This is a flowchart of a method for maintaining a memory index according to an embodiment of the present invention. S131 includes: S1311, the memory index is divided into multiple logical partitions, and independent memory resources are configured for each logical partition; S1312, When an update request for the fine-grained tag or the link information is received, the update request is routed to the corresponding logical partition for processing according to the file internal element identifier of the update request; S1313: Within each logical partition, concurrent updates are handled according to a lock-free data structure or an optimistic locking mechanism to reduce lock contention. S1314, Periodically write the fine-grained tags and the link information to the persistent storage medium in the metadata independent hot zone in an incremental manner; S1315, during system startup or fault recovery, the fine-grained tag and the link information are loaded from the persistent storage medium into the memory index.

[0057] Specifically, dividing the in-memory index into multiple logical partitions aims to distribute the large-scale index data for storage, thereby improving the index's concurrent processing capabilities and scalability. Each logical partition is configured with independent memory resources, ensuring resource isolation between partitions, avoiding resource contention, and thus improving the processing efficiency and stability of a single partition.

[0058] When an update request for fine-grained tags or links is received, the system intelligently routes the update request to the corresponding logical partition for processing based on the file element identifiers contained in the update request. This routing mechanism ensures that the update operation can accurately apply to the logical partition where the relevant data is located, effectively avoiding the use of global locks and reducing the latency of the update operation.

[0059] In practical applications, within each logical partition, lock-free data structures or optimistic locking mechanisms can be used to efficiently handle concurrent updates. Lock-free data structures achieve concurrency control through atomic operations, avoiding the overhead and deadlock risk of traditional locking mechanisms; optimistic locking mechanisms allow concurrent operations, checking for conflicts only upon commit, and performing rollbacks or retrying if conflicts occur, thereby significantly reducing lock contention and improving system throughput.

[0060] Furthermore, to ensure data persistence and recoverability, fine-grained tags and link information are periodically and incrementally written to persistent storage media in a metadata-separate hotspot. This incremental writing method reduces the amount of data written each time, minimizing the impact on system performance, while ensuring that the latest state of the data is saved in a timely manner.

[0061] When the system needs to be restarted or recover from a failure, fine-grained tags and linking information can be loaded from the persistent storage medium into the memory index. By loading data from the persistent storage medium, the state of the memory index can be quickly restored, ensuring data consistency and business continuity in abnormal situations.

[0062] The proposed solution effectively distributes the pressure of data storage and processing by logically partitioning the memory index and configuring independent memory resources for each partition, thereby improving the overall scalability of the index.

[0063] In some embodiments described above, a local graph relational database is proposed to be deployed within the processing module to handle semantic association information. However, in practical applications, the semantic association information of archives may exhibit significant differences in access frequency and association strength. If all semantic data adopts a uniform storage and query strategy, it may result in low efficiency for frequently accessed key semantic queries, or excessive consumption of high-performance storage resources for infrequently accessed non-key semantic data, thereby affecting the overall system performance and resource utilization.

[0064] In response, this application further proposes deploying a local graph relation database based on semantic association information, including: The graph structure of the local graph relational database is divided into a core semantic layer and an extended semantic layer; The core semantic layer stores frequently accessed and strongly related semantic nodes and edges, and adopts a memory-optimized graph storage structure. The extended semantic layer stores frequently accessed and weakly associated semantic nodes and edges, and employs a disk-optimized graph storage structure. When a semantic association query request is received, graph traversal and query are performed in the core semantic layer to obtain preliminary query results; When the preliminary query results are incomplete, a supplementary query is performed in the extended semantic layer according to the extended requirements of the semantic association query request to obtain supplementary query results. The preliminary query results and the supplementary query results are then integrated to obtain the final query results.

[0065] Specifically, the graph structure of the local graph relational database is divided into a core semantic layer and an extended semantic layer, aiming to perform hierarchical storage based on the access frequency and correlation strength of semantic data. The core semantic layer is designed to store frequently accessed and strongly correlated semantic nodes and edges. This data typically represents the most core and frequently queried parts of the archival information, such as the main reference relationships between files, and the connections between key figures or events. To ensure rapid response to this critical data, the core semantic layer employs a memory-optimized graph storage structure, such as based on in-memory graph database technology, to minimize I / O latency and provide millisecond-level query performance.

[0066] Furthermore, the extended semantic layer is used to store semantic nodes and edges that are accessed infrequently but still have some relevance. This data may include some auxiliary, less frequently queried, but potentially needed information for in-depth analysis. Considering its low access frequency, the extended semantic layer adopts a disk-optimized graph storage structure, such as a graph database based on persistent storage media like SSDs or HDDs, to effectively control storage costs while ensuring data integrity.

[0067] In practical applications, when a system receives a semantic association query request, it first performs graph traversal and querying at the core semantic layer. This is because most query requests typically involve frequently accessed core semantic data; by prioritizing the query in the core semantic layer in memory, preliminary query results can be quickly obtained. If the preliminary query results already meet the requirements of the query request, no further operations are needed, thus greatly improving query efficiency.

[0068] However, when the initial query results are incomplete—that is, the core semantic layer fails to provide all the necessary related information—the system will perform supplementary queries at the extended semantic layer based on the expanded requirements of the semantic association query request. This means the system intelligently determines whether further exploration of low-frequency related data is needed. After the supplementary queries are completed, the initial and supplementary query results are integrated to obtain the final query results. This hierarchical query strategy ensures that high-performance storage is prioritized while maintaining query completeness, avoiding unnecessary full-graph scans, thereby optimizing the query path and resource consumption.

[0069] The proposed solution divides the graph structure of a local graph relational database into a core semantic layer and an extended semantic layer, and adopts a hierarchical query strategy. This effectively solves the performance bottleneck and resource waste problems that exist in traditional single graph databases when processing semantic data with different access frequencies and association strengths.

[0070] In some preferred embodiments, suppose a digital archival system needs to manage semantic relationships between a large number of archives. For example, archive A references archive B, archive B mentions person C, person C participates in event D, and event D is related to archive E. The reference relationship between archives A and B, and the association between person C and event D, are the core semantics of high-frequency queries. The association between archive E and event D may belong to the extended semantics of low-frequency queries.

[0071] Specifically, when a query request needs to find "all files and people directly related to file A", the system will first perform a query at the core semantic layer. Since the reference relationship between file A and file B, and the mention relationship between file B and person C are stored in the core semantic layer, the system can quickly return file B and person C as preliminary query results.

[0072] If the query request is further expanded to "find all events indirectly related to file A", and the association between event D and file E is determined to be extended semantics, it is stored in the extended semantic layer. At this point, the initial query results (file B, character C) are insufficient to meet the needs. The system will perform supplementary queries in the extended semantic layer based on the extended requirements, find event D in which character C is involved, and further discover the association between event D and file E. Finally, the system integrates the query results from the core semantic layer and the extended semantic layer, returning to the user all files, characters, and events directly and indirectly related to file A. This layered processing approach ensures a fast response to high-frequency queries, while also providing comprehensive deep association queries when needed, effectively balancing performance and storage costs.

[0073] In some embodiments described above, this application proposes maintaining a semantic information index related to local archives through a processing module on each data shard. This includes maintaining an inverted text index based on keyword and tag information. However, in actual implementation, facing massive amounts of archive data and diverse query requirements, if the inverted text index is not managed meticulously, it may lead to problems such as low index update efficiency, limited query performance, and wasted storage resources. For example, if high-frequency and low-frequency keywords are stored and managed using the same strategy, it will be difficult to balance query response speed and storage cost. Failure to address these issues will affect the efficiency and scalability of the entire archive information storage system. Therefore, this application further proposes an optimized method for maintaining the inverted text index. Through mechanisms such as hierarchical storage, activity routing, and content-sensitive compression, it effectively improves the index's performance and resource utilization.

[0074] In response, this application further proposes maintaining an inverted text index based on keyword and tag information, including: The inverted text index is divided into a core index layer and an extended index layer; The core index layer stores frequently accessed keyword and tag information, and uses a memory-optimized data structure; The extended index layer stores frequently accessed keyword and tag information and employs a disk-optimized data structure. When an update request for the keyword or tag information is received, the update request is routed to the core index layer or the extended index layer for updating based on the activity level of the keyword or tag in the update request. When a query request is received, a query is performed in the core index layer to obtain preliminary query results; if the preliminary query results are incomplete, a supplementary query is performed in the extended index layer to obtain supplementary query results, and the preliminary query results and the supplementary query results are integrated to obtain the final query results; Content-sensitive inverted list compression technology is used to compress the inverted list in the inverted text index.

[0075] Specifically, the inverted text index is divided into a core index layer and an extended index layer, designed for hierarchical storage based on the access frequency and importance of keyword and tag information. The core index layer is designed to store frequently accessed keywords and tags with high real-time requirements. To ensure fast response, this core index layer typically employs memory-optimized data structures such as hash tables, B-trees, or Trie trees to minimize disk I / O and improve query and update efficiency. The extended index layer stores infrequently accessed, historical, or archived keyword and tag information. Considering storage costs and data volume, this extended index layer typically employs disk-optimized data structures such as B+ trees, LSM trees, or columnar storage to achieve efficient disk storage and retrieval.

[0076] When the system receives an update request for keyword or tag information, it determines the routing path based on the activity level of that keyword or tag. Activity level can be understood as the frequency with which the keyword or tag is accessed or modified within a certain time window. For example, activity level can be calculated by counting access counts, update counts, or using a time decay function. If the activity level of a keyword or tag is high, the update request will be routed to the core index layer for processing to ensure fast response and data consistency. Conversely, if the activity level is low, the update request will be routed to the extended index layer. This activity level-based routing mechanism effectively balances the performance requirements of different data access patterns.

[0077] When processing query requests, the system first performs a query in the core index layer. Since the core index layer stores frequently accessed data and uses a memory-optimized structure, preliminary query results can be obtained quickly. If the preliminary query results already meet the query requirements, no further action is needed. However, when the preliminary query results are incomplete, i.e., the query scope may involve infrequently accessed data, the system performs a supplementary query in the extended index layer. After the supplementary query is completed, the preliminary query results and the supplementary query results are integrated to obtain the final complete query result. This hierarchical query strategy prioritizes the performance requirements of high-frequency queries while ensuring the comprehensiveness of the query.

[0078] Furthermore, this application introduces a content-sensitivity-based inverted index compression technique to compress the inverted lists within the inverted text index. Content sensitivity refers to attributes such as the importance, confidentiality level, or access restrictions of the archive content. By combining content sensitivity with compression strategies, more refined storage optimization can be achieved. For example, for inverted lists containing highly sensitive content, a lower compression ratio or lossless compression algorithm can be used to ensure data integrity and fast decompression; while for inverted lists containing non-sensitive content, a high compression ratio algorithm can be used to maximize storage space savings.

[0079] The proposed solution divides the inverted text index into a core index layer and an extended index layer, and updates the routing based on the activity of keywords and tags. This effectively solves the performance bottleneck and resource waste problems faced by traditional single index structures when processing massive amounts of archival data.

[0080] In some of the embodiments described above in this application, a content-sensitivity-based inverted index compression technique was proposed. However, in its implementation, if the dynamic variability of the sensitivity of archival content and the differentiated needs of data with different sensitivity levels during storage and retrieval are not fully considered, it may lead to poor compression efficiency, low resource utilization, and even performance bottlenecks when handling complex queries across sensitivity levels. If these problems are not addressed, the overall efficiency and security of archival information storage may be affected. Therefore, this application further proposes a more refined content-sensitivity-based inverted index compression method, aiming to adapt to the diverse sensitivity characteristics of archival content by dynamically adjusting the compression strategy, distinguishing the storage of sensitive data, and optimizing the query processing flow.

[0081] In response, this application further proposes the aforementioned content-sensitive inverted list compression technique, the steps of which include compressing the inverted list in the inverted text index: Obtain sensitive information about the contents of the archives; Based on the sensitivity information and preset compression algorithm performance parameters, a compression algorithm is dynamically selected to compress the inverted list in the inverted text index; When the sensitivity information of the archive content changes, the sensitivity level of the archive content is reassessed, and the compression algorithm is dynamically adjusted according to the new sensitivity level and the preset compression algorithm performance parameters. When compressing the inverted list in the inverted text index, the sensitivity information of the archive content is embedded into the compressed data structure; Before compressing the inverted list in the inverted text index, the entries in the inverted list are divided into sensitive entries and non-sensitive entries according to the sensitivity level of the file content, and stored separately. When a query request is received that crosses fragments and sensitivity levels, the inverted list of different compression strategies is identified before decompression based on the embedded sensitivity information. When merging and deduplicating inverted lists using different compression strategies, a unified intermediate data format is used for conversion, and merging and deduplication operations are performed after conversion.

[0082] Specifically, obtaining sensitivity information about archival content refers to the system identifying and quantifying the sensitivity of archival data through predefined rules, machine learning models, or manual annotation. For example, the sensitivity level can be determined based on factors such as the archive's classification, content type (e.g., personal privacy, trade secrets), or access permissions. The purpose is to provide a basis for subsequent differentiated compression and storage.

[0083] The system dynamically selects a compression algorithm based on sensitivity information and preset compression algorithm performance parameters. This can be understood as the system intelligently choosing the most suitable compression algorithm for the current file content based on its sensitivity level (e.g., high sensitivity, medium sensitivity, low sensitivity) and pre-configured performance metrics of various compression algorithms (e.g., compression ratio, decompression speed, CPU / memory consumption). For example, for highly sensitive data, an algorithm with high security but a slightly lower compression ratio or speed might be prioritized; for low-sensitivity data, an algorithm with a high compression ratio and fast speed might be selected. The goal is to maximize storage efficiency and query performance while ensuring data security.

[0084] In practical applications, when the sensitivity information of archival content changes, such as due to policy adjustments or data reclassification, the system automatically triggers a reassessment process. Based on the new sensitivity level and preset compression algorithm performance parameters, it dynamically adjusts the compression algorithm for stored or to-be-stored data. The purpose is to ensure that the compression strategy remains consistent with the latest sensitivity status of the archives.

[0085] Furthermore, when compressing the inverted list in the inverted text index, embedding the sensitivity information of the archive content into the compressed data structure means explicitly including the sensitivity level identifier of the archive content corresponding to the data block in the metadata or header information of the compressed data block. The purpose is to quickly identify its sensitivity attributes without decompressing the entire data block, thereby enabling efficient filtering and processing based on sensitivity during queries.

[0086] Furthermore, before compressing the inverted list in the inverted text index, the entries in the inverted list are divided into sensitive and non-sensitive entries based on the sensitivity level of the file content, and stored separately. This can be understood as logically or physically separating the entries in the inverted list according to the sensitivity of their associated files. For example, all entries associated with highly sensitive files can be stored in a separate inverted list partition, while entries associated with non-sensitive files can be stored in another partition. The purpose is to achieve isolated storage and management of data with different sensitivities, improve security, and allow for different access control and processing strategies for data with different sensitivities.

[0087] When receiving query requests that span multiple shards and sensitivity levels, the query engine identifies inverted lists with different compression strategies before decompression based on the embedded sensitivity information. This means that when processing complex queries, the query engine can use the sensitivity identifiers embedded in the data structure to quickly locate and identify which inverted lists use which compression strategies, thereby calling the correct decompression algorithm during the decompression stage, avoiding unnecessary full decompression, and improving query efficiency.

[0088] Finally, when merging and deduplicating inverted lists using different compression strategies, a unified intermediate data format is used for conversion. Merging and deduplication are then performed after conversion. This means that to handle inverted list data from different compression strategies and sensitivity levels, the system first decompresses and converts this data into a standardized, unified intermediate data format. The purpose is to eliminate data format differences caused by different compression strategies, simplify subsequent merging, deduplication, and analysis operations, and ensure the accuracy and consistency of data processing.

[0089] The solution proposed in this application effectively addresses the efficiency and security issues that may exist in the above-mentioned basic solutions when processing diverse sensitive archival data by introducing sensitive information of the archival content and deeply integrating it into the compression, storage, and query processing flow of the inverted list.

[0090] In some of the embodiments described above in this application, a unified intermediate data format is proposed for conversion when merging and deduplicating inverted lists using different compression strategies, followed by merging and deduplication operations. However, in practical implementation, how to efficiently, in parallel, and reliably perform large-scale data merging and deduplication to avoid becoming a system performance bottleneck when facing inverted list data streams from different compression strategies and potentially across shards is a problem that needs further resolution. Especially in distributed cloud database environments, where the data volume is massive and real-time requirements are high, traditional serial processing or simple merging methods may not meet performance demands. To address this, this application further proposes an optimized merging and deduplication method, which introduces mechanisms such as block processing, parallelization, local deduplication, and final aggregation, aiming to significantly improve processing efficiency and system throughput.

[0091] To address this, this application further proposes using a unified intermediate data format for conversion when merging and deduplicating inverted lists with different compression strategies, and then performing merging and deduplication operations after conversion, including: A block processing mechanism is adopted to divide the data stream of the inverted list into multiple data blocks; Each data block is decompressed and format converted independently to obtain the conversion result; The conversion result is stored according to the local cache; The conversion result is transmitted to an intermediate storage area using an asynchronous write method. By parallelizing the processing, multiple data blocks in the intermediate storage area are distributed to different computing threads or processes; In each computing thread or process, local deduplication is performed on the received data blocks, and the processed terms and file identifiers are recorded based on a hash-based data structure; The results of local deduplication are summarized and then deduplicated to obtain the final merged deduplication result.

[0092] Specifically, the block processing mechanism refers to logically or physically dividing the original inverted list data stream into several independent data blocks according to a preset size or number of records. For example, each data block can be set to contain a fixed number of inverted records, or its size can not exceed a specific number of bytes. The purpose is to decompose large-scale data into manageable small units, facilitating parallel processing and resource scheduling.

[0093] The independent decompression and format conversion of each data block means that each data block can be processed by an independent processor or thread without waiting for other data blocks. The decompression operation aims to recover the original data, while the format conversion unifies data from different compression strategies into an intermediate data format, such as a unified JSON format or binary structure. Its purpose is to provide standardized input for subsequent merging and deduplication operations.

[0094] In practical applications, the local cache can be understood as a high-speed storage medium, such as memory or a solid-state drive, located within the processing unit, used for temporary storage of transformation results. Its purpose is to reduce frequent access to the main storage area and improve data access speed. Furthermore, using asynchronous writing to transmit the transformation results to an intermediate storage area means that after data block processing is complete, the transformation results are not immediately written to final storage, but are sent to a temporary, scalable storage area, such as a message queue or a distributed file system, in a non-blocking manner. Its purpose is to decouple data processing from data storage, improving system responsiveness and throughput.

[0095] Furthermore, through parallel processing, multiple data blocks in the intermediate storage area can be distributed to different computing threads or processes for processing. For example, the resources of multi-core CPUs or distributed computing clusters can be utilized to process multiple data blocks simultaneously. The aim is to fully utilize computing resources and significantly shorten the overall processing time. Within each computing thread or process, local deduplication of the received data blocks refers to identifying and removing duplicate terms or file identifiers within a single processing unit. This is typically based on hash-based data structures, such as Bloom filters or hash tables, to quickly determine whether a term or file identifier has already been processed. The purpose is to reduce data redundancy locally and lower the computational burden of subsequent global deduplication.

[0096] Finally, the results of local deduplication are summarized and then deduplicated again. This involves collecting the local deduplication results from all computation threads or processes and performing a final deduplication operation globally. This can be accomplished through a centralized coordinator or another distributed processing stage to ensure that all duplicates are completely removed. The goal is to guarantee that the final output inverted list is completely deduplicated and accurate.

[0097] The solution proposed in this application introduces a block processing mechanism to decompose a large-scale inverted list data stream into data blocks that are easy to manage and process in parallel, thereby effectively solving the performance bottleneck that may occur when traditional methods process massive amounts of data.

[0098] In some of the embodiments described above in this application, when performing local deduplication on the received data blocks and recording the processed terms and file identifiers based on a hash-based data structure in each computing thread or process, using only a single hash function or failing to adequately consider data activity may lead to frequent hash collisions, thereby affecting deduplication efficiency and data storage performance. Furthermore, failing to differentiate the storage of data with different activity levels may result in wasted memory resources and reduced overall system response speed.

[0099] In this regard, this application further proposes the following steps: in each computing thread or process, locally deduplicating the received data block and recording the processed terms and file identifiers based on a hash data structure, including: In each computing thread or process, when performing local deduplication on the received data block, the term or file identifier is mapped to one of multiple hash functions based on its characteristics to obtain the first mapping result; When a hash collision occurs in the first mapping result, the conflicting term or file identifier is attempted to be mapped to another preset hash function to obtain a second mapping result; If the second mapping result still results in a conflict, the conflicting term or file identifier is stored in an overflow area. When recording processed terms and file identifiers, they are stored in different hash table partitions based on the activity level of the terms or file identifiers. For highly active terms or file identifiers, allocate separate hash table partitions and use a memory-optimized hash structure.

[0100] Specifically, when performing local deduplication on received data blocks, a multi-hash strategy can be employed to effectively handle hash collisions. First, the characteristics of the term or file identifier are input into one of several preset hash functions to generate a first mapping result. For example, different hash functions can be selected based on the term's length, first letter, or specific encoding rules. When a hash collision occurs in the first mapping result—that is, when different terms or file identifiers are mapped to the same location in the hash table—the system does not immediately report the collision or overwrite data. Instead, it attempts to remap the colliding term or file identifier using another preset hash function to obtain a second mapping result. This mechanism aims to reduce the probability of collisions by providing alternative mapping paths. If a collision still occurs after the second mapping, to ensure all data is properly processed, the colliding term or file identifier is stored in a dedicated overflow area. This overflow area can be a linked list, an array, or a separate hash table, used to store data that cannot be directly located through the main hash table.

[0101] Furthermore, when recording processed terms and file identifiers, storage can be differentiated based on the activity level of the terms or file identifiers to optimize storage efficiency and access performance. Activity level can be understood as the frequency with which data is accessed or updated. For example, activity level can be assessed based on historical access logs, query frequency, or update frequency. Based on the assessment results, terms or file identifiers are stored in different hash table partitions. Specifically, terms or file identifiers determined to be highly active are assigned independent hash table partitions. These independent hash table partitions typically employ memory-optimized hash structures, such as Cuckoo hashes, cuckoo filters, or linear probing hashes, to ensure fast lookup and insertion operations, thereby improving the processing efficiency of frequently accessed data.

[0102] The proposed solution effectively reduces the probability of hash collisions that may be caused by a single hash function by introducing multiple hash functions and an overflow region mechanism, thereby improving the accuracy and efficiency of the local deduplication process.

[0103] Specifically, the above-mentioned allocation of independent hash table partitions to highly active terms or file identifiers, and the use of a memory-optimized hash structure, can be implemented in the following way.

[0104] In each computation thread or process, a separate hash table partition is assigned to highly active terms or file identifiers; Receive new terms or file identifiers and evaluate the activity of the terms or file identifiers; Based on the activity level and a preset activity threshold, determine whether the term or file identifier belongs to high-activity data; The highly active data is mapped to the corresponding hash table partition using a hash function; Within the hash table partition, a memory-optimized hash structure based on Cuckoo hashing is adopted, and the hash structure reserves a certain proportion of spare storage space during initialization; When the hash table partition approaches its capacity limit, a dynamic expansion process is initiated, which is accomplished by allocating new memory blocks in the background and gradually migrating data.

[0105] Specifically, within each computing thread or process, to effectively manage and process data, a dedicated hash table partition is allocated to highly active terms or archive identifiers. This allocation strategy aims to isolate frequently accessed and updated data from ordinary data, thereby optimizing processing efficiency. When the system receives a new term or archive identifier, it performs an activity assessment. This assessment can be based on various metrics, such as the access frequency, update frequency, and number of citations of the term or archive identifier. By setting a preset activity threshold, the system can determine whether a term or archive identifier belongs to high-activity data. For example, a threshold can be set whereby a term is considered highly active if it is accessed more than N times within a unit of time.

[0106] Furthermore, data identified as highly active is mapped to a pre-allocated independent hash table partition using a hash function. This hash function is designed to ensure a uniform distribution of data within the partition, avoiding hotspot issues. Within the hash table partition, a memory-optimized hash structure based on Cuckoo hashing is employed. Cuckoo hashing is an efficient hash algorithm that allows each key-value pair to be stored in one of multiple possible hash locations and resolves collisions by "kicking out" existing elements, thus achieving high lookup efficiency and space utilization in memory. As a preferred implementation, the hash structure reserves a certain percentage of spare storage space during initialization. This spare space is used to cope with sudden data growth or hash collisions, reducing the need for frequent expansion during runtime, thereby improving system stability and performance.

[0107] Furthermore, to cope with the continuous growth of data volume, a dynamic expansion process is initiated when the hash table partition approaches its capacity limit. This dynamic expansion process is accomplished by allocating new memory blocks in the background and gradually migrating data. This background expansion mechanism can minimize the impact on front-end operations, ensuring the continuity and availability of data processing. For example, a dual-write strategy can be adopted, writing data to both the old and new partitions simultaneously, and then switching to the new partition and releasing the resources of the old partition after the data migration is complete.

[0108] This application's solution achieves refined management of highly active data by evaluating the activity of terms or archive identifiers and routing them to independent hash table partitions based on the evaluation results. Through this technical solution, this application enables more efficient and stable management of highly active terms or archive identifiers. Specifically, the independent hash table partitions, combined with the Cuckoo hash structure, significantly improve the access speed and concurrent processing capabilities of highly active data, effectively reducing the probability of lock contention and hash collisions. The introduction of reserved backup storage space and a dynamic expansion mechanism allows the system to maintain consistently high performance when facing data volume fluctuations, avoiding performance bottlenecks caused by frequent expansions or insufficient capacity. Compared to simply allocating hash table partitions, this application, through meticulous activity evaluation, intelligent hash mapping, optimized memory structure, and dynamic expansion strategies, ensures the real-time performance and reliability of data processing, thus providing a more robust and efficient deduplication mechanism for the storage and processing of digitized archival information in a distributed cloud database environment.

[0109] refer to Figure 4 , Figure 4 This is a schematic diagram of a cloud database-based digital archival information storage system provided by an embodiment of the present invention. It is used to process archival description information updates and complex relational queries in a distributed cloud database environment where archival data is fragmented according to creation time ranges, including: The input terminal is used to receive a query request containing multiple query conditions and decompose the query request into multiple independent query parts; according to the data characteristics of each query part, the query part is distributed to the corresponding processing unit. The processing end is used to deploy a processing module on each data shard. The processing module maintains a semantic information index related to local archives and performs preliminary screening of the local archives based on the received query part and the semantic information index to obtain concise archive identification information. The output end is used to aggregate file identification information from multiple processing modules and obtain file content based on the aggregated file identification information.

[0110] This system intelligently receives and decomposes complex query requests through its input end, and distributes the query portions to the corresponding processing units based on data characteristics. The processing modules deployed on each data shard utilize local semantic information indexes to perform preliminary screening of archives, thereby efficiently generating concise archive identification information. Finally, the output end is responsible for aggregating this concise identification information and retrieving the archive content as needed. This architectural design effectively solves the problem of low efficiency in handling complex relational queries in traditional distributed cloud databases, significantly improving query response speed and user experience, and ensuring efficient management and retrieval in environments with massive amounts of digital archive data.

[0111] The core of the cloud database-based digital archival information storage system proposed in this application lies in the intelligent processing and distributed execution of complex query requests.

[0112] Specifically, the input endpoint is configured to receive query requests containing multiple query conditions and decompose the query requests into multiple independent query parts; based on the data characteristics of each query part, the query parts are distributed to the corresponding processing units. For example, the input endpoint could be a query parsing service deployed on a gateway server, responsible for receiving raw query requests from clients. This service includes a query parser that identifies logical operators and different query predicates in the query string, thereby decomposing a complex query into smaller, independently executable sub-query units. Subsequently, a query router routes the query to a pre-configured processing unit based on the hash value of specific fields contained in the query part, the query type, or the target data sharding information. As a preferred implementation, the input endpoint could consist of a load balancer and a set of stateless API gateways, which are responsible for receiving requests and forwarding them to the internal query decomposition service. In another implementation, the input endpoint could be a query processing component integrated into an application server, performing preliminary query parsing and distribution directly at the application layer.

[0113] Furthermore, the processing end is configured to deploy a processing module on each data shard. This processing module maintains a semantic information index related to local archives and performs preliminary filtering of the local archives based on the received query portion and the semantic information index to obtain concise archive identification information. For example, the processing end can be a lightweight service running on each data shard server, listening for query portions distributed from the input end. The processing module integrates various local index structures, such as an inverted index for keyword matching, a graph relational database for semantic association, and an in-memory index for fine-grained tags. When a query portion is received, the processing module uses these local indexes to quickly filter the archive metadata stored on that shard and returns concise archive identification information. As a preferred implementation, the processing module can be a containerized microservice, dynamically deployed and managed on each data shard node through a container orchestration platform (such as Kubernetes). In another implementation, the processing module can be a data processing plugin tightly integrated with a database engine, directly filtering data at the database level.

[0114] Furthermore, the output endpoint is configured to aggregate file identification information from multiple processing modules and retrieve file content based on the aggregated file identification information. For example, the output endpoint can be a central coordinator or aggregation service that collects file identification information returned by processing modules from all relevant data shards. Since this identification information is concise, transmission overhead is low. After aggregation, the output endpoint retrieves complete file content from the storage layer (e.g., object storage or file system) as needed based on this file identification information. For example, an aggregation service can receive a list of file IDs from different processing modules, then deduplicate and sort these IDs, and finally initiate batch content retrieval requests to the underlying storage system based on these unique file IDs. As a preferred implementation, the output endpoint can be a distributed message queue system used to collect the filtering results from the processing modules, and these messages are consumed by an independent aggregation service for final processing. In another implementation, the output endpoint can be an event-driven function computation service that automatically triggers content retrieval logic when file identification information is received.

[0115] The digital archival information storage system based on cloud database proposed in this application can effectively solve the challenges faced by traditional distributed cloud databases in handling complex relational queries through the synergistic effect of the above components.

[0116] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for storing digital archival information based on a cloud database, used in a distributed cloud database environment where archival data is fragmented according to creation time range, to handle updates of archival description information and complex relational queries, characterized in that, include: Receive a query request containing multiple query conditions, and decompose the query request into multiple independent query parts; Based on the data characteristics of each query part, the query part is distributed to the corresponding processing unit; On each data shard, a processing module is deployed. The processing module maintains a semantic information index related to local archives and performs preliminary screening of the local archives based on the received query and the semantic information index to obtain concise archive identification information. The system aggregates file identification information from multiple processing modules and obtains file content based on the aggregated file identification information.

2. The method for storing digital archival information based on a cloud database according to claim 1, characterized in that, On each data shard, a processing module is deployed. This module maintains a semantic information index related to local archives and performs preliminary filtering of the local archives based on the received query and the local semantic information index to obtain concise archive identification information, including: Within the processing module, a memory index is maintained based on fine-grained tags and link information; Deploy a local graph relation database based on semantic association information; Maintain an inverted text index based on keyword and tag information; A routing and transformation layer is set up to coordinate the updates and queries of the memory index, the local graph relational database, and the inverted text index; On each data shard, semantic metadata and its index are stored in an independent metadata hotspot. When the query requires the original content of the file, the content on-demand activation mechanism is activated to retrieve the file content from the low-cost storage layer.

3. The method for storing digital archival information based on a cloud database according to claim 2, characterized in that, Within the processing module, a memory index is maintained based on fine-grained tags and link information, including: The memory index is divided into multiple logical partitions, and each logical partition is configured with independent memory resources; When an update request for the fine-grained tag or the link information is received, the update request is routed to the corresponding logical partition for processing based on the file internal element identifier of the update request; Within each logical partition, concurrent updates are handled using lock-free data structures or optimistic locking mechanisms to reduce lock contention. The fine-grained tags and the link information are periodically written incrementally to the persistent storage medium in the metadata independent hot zone; During system startup or fault recovery, the fine-grained tags and the linking information are loaded from the persistent storage medium into the memory index.

4. The method for storing digital archival information based on a cloud database according to claim 2, characterized in that, The step of deploying a local graph relation database based on semantic association information includes: The graph structure of the local graph relational database is divided into a core semantic layer and an extended semantic layer; The core semantic layer stores frequently accessed and strongly related semantic nodes and edges, and adopts a memory-optimized graph storage structure. The extended semantic layer stores frequently accessed and weakly associated semantic nodes and edges, and employs a disk-optimized graph storage structure. When a semantic association query request is received, graph traversal and query are performed in the core semantic layer to obtain preliminary query results; When the preliminary query results are incomplete, a supplementary query is performed in the extended semantic layer according to the extended requirements of the semantic association query request to obtain supplementary query results. The preliminary query results and the supplementary query results are then integrated to obtain the final query results.

5. A method for storing digital archival information based on a cloud database according to claim 2, characterized in that, The process of maintaining an inverted text index based on keyword and tag information includes: The inverted text index is divided into a core index layer and an extended index layer; The core index layer stores frequently accessed keyword and tag information, and uses a memory-optimized data structure; The extended index layer stores frequently accessed keyword and tag information and employs a disk-optimized data structure. When an update request for the keyword or tag information is received, the update request is routed to the core index layer or the extended index layer for updating based on the activity level of the keyword or tag in the update request. When a query request is received, a query is performed in the core index layer to obtain preliminary query results; if the preliminary query results are incomplete, a supplementary query is performed in the extended index layer to obtain supplementary query results, and the preliminary query results and the supplementary query results are integrated to obtain the final query results; Content-sensitive inverted list compression technology is used to compress the inverted list in the inverted text index.

6. The method for storing digital archival information based on a cloud database according to claim 5, characterized in that, The content-sensitive inverted list compression technology compresses the inverted list in the inverted text index, including: Obtain sensitive information about the contents of the archives; Based on the sensitivity information and preset compression algorithm performance parameters, a compression algorithm is dynamically selected to compress the inverted list in the inverted text index; When the sensitivity information of the archive content changes, the sensitivity level of the archive content is reassessed, and the compression algorithm is dynamically adjusted according to the new sensitivity level and the preset compression algorithm performance parameters. When compressing the inverted list in the inverted text index, the sensitivity information of the archive content is embedded into the compressed data structure; Before compressing the inverted list in the inverted text index, the entries in the inverted list are divided into sensitive entries and non-sensitive entries according to the sensitivity level of the file content, and stored separately. When a query request is received that crosses fragments and sensitivity levels, the inverted list of different compression strategies is identified before decompression based on the embedded sensitivity information. When merging and deduplicating inverted lists using different compression strategies, a unified intermediate data format is used for conversion, and merging and deduplication operations are performed after conversion.

7. A method for storing digital archival information based on a cloud database according to claim 6, characterized in that, When merging and deduplicating inverted lists using different compression strategies, a unified intermediate data format is used for conversion, followed by merging and deduplication operations, including: A block processing mechanism is adopted to divide the data stream of the inverted list into multiple data blocks; Each data block is decompressed and format converted independently to obtain the conversion result; The conversion result is stored according to the local cache; The conversion result is transmitted to an intermediate storage area using an asynchronous write method. By parallelizing the processing, multiple data blocks in the intermediate storage area are distributed to different computing threads or processes; In each computing thread or process, local deduplication is performed on the received data blocks, and the processed terms and file identifiers are recorded based on a hash-based data structure; The results of local deduplication are summarized and then deduplicated to obtain the final merged deduplication result.

8. A method for storing digital archival information based on a cloud database according to claim 7, characterized in that, In each computing thread or process, local deduplication is performed on the received data blocks, and based on a hash-based data structure, the processed terms and file identifiers are recorded, including: In each computing thread or process, when performing local deduplication on the received data block, the term or file identifier is mapped to one of multiple hash functions based on its characteristics to obtain the first mapping result; When a hash collision occurs in the first mapping result, the conflicting term or file identifier is attempted to be mapped to another preset hash function to obtain a second mapping result; If the second mapping result still results in a conflict, the conflicting term or file identifier is stored in an overflow area. When recording processed terms and file identifiers, they are stored in different hash table partitions based on the activity level of the terms or file identifiers. For highly active terms or file identifiers, allocate separate hash table partitions and use a memory-optimized hash structure.

9. A method for storing digital archival information based on a cloud database according to claim 8, characterized in that, The method of allocating independent hash table partitions for highly active terms or file identifiers and employing a memory-optimized hash structure includes: In each computation thread or process, a separate hash table partition is assigned to highly active terms or file identifiers; Receive new terms or file identifiers and evaluate the activity of the terms or file identifiers; Based on the activity level and a preset activity threshold, determine whether the term or file identifier belongs to high-activity data; The highly active data is mapped to the corresponding hash table partition using a hash function; Within the hash table partition, a memory-optimized hash structure based on Cuckoo hashing is adopted, and the hash structure reserves a certain proportion of spare storage space during initialization; When the hash table partition approaches its capacity limit, a dynamic expansion process is initiated, which is accomplished by allocating new memory blocks in the background and gradually migrating data.

10. A digital archival information storage system based on a cloud database, used to process archival description information updates and complex relational queries in a distributed cloud database environment where archival data is fragmented according to creation time range, characterized in that, include: The input terminal is used to receive a query request containing multiple query conditions and decompose the query request into multiple independent query parts; Based on the data characteristics of each query part, the query part is distributed to the corresponding processing unit; The processing end is used to deploy a processing module on each data shard. The processing module maintains a semantic information index related to local archives and performs preliminary screening of the local archives based on the received query part and the semantic information index to obtain concise archive identification information. The output end is used to aggregate file identification information from multiple processing modules and obtain file content based on the aggregated file identification information.

Images