Office knowledge base-oriented cloud document deduplication archiving and trace management method and system

Abstract

This invention discloses a method and system for deduplication, archiving, and traceability management of cloud documents for office knowledge bases, relating to the field of semantic processing technology. The method and system include the following steps: S1, event-driven data collection of document objects to obtain the original dataset and perform normalization processing; S2, generating multi-layer fingerprints, performing candidate duplicate retrieval and alignment, and evaluating archiving duplication; S3, generating differential units and evaluating intent strength to construct an indexing strategy; S4, constructing a version causal relationship network, generating archiving evidence packages, and performing consistency verification. This invention effectively improves the accuracy of deduplication and archiving of office knowledge base cloud documents, archiving throughput efficiency, and version traceability reliability, solving the problem that document version differential algorithms generate redundant difference descriptions for complex editing operations and are difficult to reverse merge.

Description

Technical Field

[0001] This invention relates to the field of semantic processing technology, specifically to a cloud-based document deduplication, archiving, and traceability management method and system for office knowledge bases. Background Technology

[0002] With the widespread use of cloud document collaboration platforms, cloud storage directories, email attachments, instant messaging files, and exported meeting minutes in office scenarios, documents are increasingly being copied, updated, and linked across systems, resulting in multiple copies, versions, and dependencies. Current technologies often rely on single-layer features or single similarity scores for deduplication, failing to adequately capture complex scenarios such as "identical content but different structures," "consistent structures but changed dependencies," and "changes in permission-sharing status," leading to incorrect merging or omissions.

[0003] For example, invention patent CN117472981B discloses a business chain tracing and retrieval method, including: S1, communicating and confirming business processes; S2, sorting out the business chain; S201, sorting from a business perspective; S202, sorting from a technical perspective; S3, configuring the business chain; and S4, tracing and retrieving the business chain. This invention transforms isolated retrieval of archived data into business tracing retrieval between different data sets. This retrieval mode primarily addresses the issue of convenient utilization of archived data from different business systems or modules. It establishes a management model that conforms to the characteristics of business processes. It solves the problem of how to long-term and securely preserve valuable data resources in enterprise ERP systems, designing a data collection, archiving, and management solution for ERP systems based on long-term preservation and authenticity management, suitable for my country's national conditions. It enables electronic archives to support various services such as auditing and technical inquiries, while also considering providing more valuable services to enterprise users.

[0004] For example, invention patent CN117312237B discloses a method, device, and medium for archiving blockchain block data, including: determining the start and end numbers of the block data to be archived; constructing an archive request to be executed using the private key of the blockchain node corresponding to the block data to be archived; publishing the archive request to be executed on the blockchain; the archiving smart contract in the blockchain reviews the archive request to be executed and publishes the archiving review result; if the review is passed, the blockchain node to be archived packages the block data to be archived to generate an archive data file and uploads it to a preset storage area; after receiving a message notification of successful upload, constructing a successful archive notification based on the archive request transaction hash corresponding to the archive request to be executed and the storage address of the archive data file to be archived contained in the message notification, and publishing it. This application avoids the risk of data loss or leakage and ensures the traceability of archiving behavior through the above method.

[0005] In existing technologies, when users perform structural edits such as moving paragraph blocks, rearranging chapters, adjusting slide order, or migrating charts across locations, fine-grained differential processing breaks down the movement into large deletions and additions, resulting in a large volume of differential records that are difficult to express the intentions of higher-level operations. This leads to difficulties in understanding differential visualization, conflict localization and automatic merging, increased costs for selective rollback and traceability chain reconstruction, and impacts auditing and collaboration efficiency.

[0006] Therefore, in response to the above problems, there is an urgent need for cloud-based document deduplication, archiving, and traceability management methods and systems for office knowledge bases. Summary of the Invention

[0007] Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a cloud-based document deduplication, archiving, and traceability management method and system for office knowledge bases, which solves the problem that document version difference algorithms generate redundant difference descriptions for complex editing operations and are difficult to reverse merge.

[0009] Technical solution

[0010] To achieve the above objectives, this invention provides the following technical solution: a cloud document deduplication archiving and traceability management method for office knowledge bases, comprising: S1, performing event-driven data collection on document objects, obtaining the original dataset and performing structured parsing to obtain a full data stream, and performing normalization and anomaly marking on the full data stream; S2, generating multi-layer fingerprints based on the full data stream, performing candidate duplicate retrieval and alignment, and evaluating the archive duplication rate, filtering duplicate candidate documents, and establishing a version chain entry point; S3, generating differential units based on the version chain entry point and evaluating intent strength, storing the differential unit data, and constructing an indexing strategy; S4, constructing a version causal relationship network, generating an archive evidence package, performing consistency verification, and outputting the cloud document deduplication archiving results.

[0011] Furthermore, event-driven data collection is performed on document objects to obtain the raw dataset and perform structured parsing to obtain the full data stream. The specific process of normalizing and anomaly marking the full data stream is as follows: Event-driven data collection is performed on document objects from the office knowledge base, including those from cloud document collaboration platforms, cloud drive directories, email attachments, instant messaging files, and exported meeting minutes, to obtain the raw dataset. The raw dataset includes text content, heading levels, table objects, image objects, attachment references, comments and annotations, and permission sharing status. Simultaneously, the collection timestamp, source system identifier, document unique identifier, and version number are recorded. The system synchronously accesses the index view of the historical archive as a comparison benchmark input. The historical archive is used to store the main document record, version chain record, candidate document, differential record, and multi-level fingerprint index of archived documents. The system performs structured parsing and unified encapsulation on the original dataset to generate a full data stream. The system performs normalization processing on the full data stream using a caliber dictionary mapping. The system writes anomaly markers and isolates the full data stream that has missing text, failed parsing, insufficient permissions, or corrupted objects. The system writes the normalized document frames with anomaly markers to the archive buffer and binds them to the candidate main document entry key in the historical archive.

[0012] Furthermore, the specific process of generating multi-layer fingerprints based on the full data stream for candidate duplicate retrieval and alignment is as follows: First, multi-layer fingerprints are generated based on the full data stream. Then, based on the text content field in the full data stream, text normalization is performed, followed by encrypted hashing of the normalized text string to obtain the content fingerprint. Next, based on the set of text blocks segmented by paragraph blocks and the block-level position index in the full data stream, the normalized line break boundaries are used as the basic segmentation points, and the text is further segmented by combining the heading level and style segment boundaries. Each paragraph block is then encrypted and hashed using SHA-256 to obtain the paragraph block fingerprint. Finally, the paragraph block fingerprints are sorted by position index to form a sequence, and Merkle tree aggregation is used to obtain the block-level fingerprint. The leaf nodes of the Merkle tree are the paragraph block fingerprints sorted by position index. The parent node is obtained by concatenating the fingerprints of two adjacent child nodes in a fixed order and then performing SM3 encrypted hashing. Based on the title hierarchy, directory structure, and object list in the full data stream, structured serialization is performed, and encrypted hashing is performed to obtain the structure fingerprint. Based on the reference relationships and attachment list established in the full data stream, the reference edges are normalized and sorted according to the priority of the reference object type, the lexicographical order of the reference object identifier, and the ascending order of the occurrence sequence number. The dependency fingerprint is used as a clear distinguishing point, and encrypted hashing is performed to obtain the dependency fingerprint. Candidate duplicate retrieval and alignment are performed within the multi-level fingerprint. Candidate documents in the historical archive are retrieved using the content fingerprint and block-level fingerprint as the primary keys. Block-level alignment is performed between the candidate documents and the current document to obtain the block overlap ratio and structural consistency, and the duplicate candidate set is output.

[0013] Furthermore, the specific process for evaluating archive duplication is as follows: the similarity between the current document to be archived and the candidate documents at four levels—content fingerprint, block-level fingerprint, structural fingerprint, and dependency fingerprint—is calculated using a consistency metric function; the content similarity, block-level similarity, structural similarity, and dependency similarity are multiplied together and then the fourth root is taken to obtain the archive duplication evaluation value, and deduplication archiving is performed.

[0014] Furthermore, the specific process of filtering duplicate candidate documents and establishing a version chain entry is as follows: The archive duplication evaluation value and the deduplication threshold are compared in real time. When the archive duplication evaluation value is not less than the deduplication threshold, the current document is merged into the main document version chain, written into the version chain incremental record set, and evidence package entries and causal network edge records corresponding to the version are generated. Differential units, references, and permission changes are written to the archive library incremental area, without repeatedly writing the complete text, to avoid redundancy but ensure the version chain and causal network remain unbroken. When the archive duplication evaluation value is less than the deduplication threshold, the current document is entered into the library as a new main document, and a version chain entry is established. Simultaneously, the main document evidence package and causal network root node are initialized.

[0015] Furthermore, the specific process of generating differential units based on the version chain entry and evaluating intent strength is as follows: For two adjacent versions merged into the same main document version chain, the block-level alignment result and object reference relationship are called; block-level differentiation is performed on the text paragraph block and object list; new blocks, deleted blocks, replaced blocks, rearranged blocks, differential types, and object reference change types are identified; differential units are generated, and each differential unit is bound to a block-level position index and source version number; operation intent annotation is constructed based on the differential units: continuous differential units are aggregated into editing sessions according to the same author, the same time window, and the same location neighborhood; editing intent labels are output according to differential types within the editing session; the operation type weight corresponding to the differential unit is obtained through evidence scoring; the change in structural depth is obtained by comparing the absolute value of the change in the hierarchical depth of the document outline before and after the editing session; the maximum structural depth is obtained by analyzing the title hierarchy structure tree of the document version before editing; and the maximum structural depth is obtained by statistically analyzing the changes in the document hierarchy structure tree before editing. The number of external references is obtained by counting the number of differential units in this editing session where all external references are added, deleted, or modified. The number of permission changes is obtained by counting the number of differential units in this editing session where all document changes occur. The differential unit index is obtained by traversing all differential units in this editing session. For each differential unit in an editing session, the operation type weight of the differential unit is multiplied by the structural depth influence coefficient to obtain the basic contribution value. The structural depth change is incremented by one and then logarithmized, then the logarithm is divided by the maximum structural depth of the document, incremented by one and then logarithmized, and then incremented by one to obtain the adjustment factor. The basic contribution value is multiplied by the adjustment factor and then summed to obtain the total contribution score. The reference change coefficient is multiplied by the number of external references to obtain the reference bonus item, and the permission change coefficient is multiplied by the number of permission changes to obtain the permission bonus item. The total contribution score is added to the reference bonus item and the permission bonus item to obtain the intent intensity evaluation value.

[0016] Furthermore, the specific process of storing differential cell data and building an index strategy is as follows: For editing sessions where the intent intensity assessment value is greater than the intent threshold, the original text of the content block involved and the snapshot of the embedded object are fully preserved in the differential cell; for editing sessions where the intent intensity assessment value is less than or equal to the intent threshold, only the content fingerprint is preserved to optimize storage; the intent intensity assessment value is written as a dynamic weight factor into the document field weight of the inverted index.

[0017] Furthermore, the specific process of constructing a version causal relationship network and generating archived evidence packages is as follows: A traceable causal graph is constructed based on version chain entry points, differential units, and operation intent annotations: versions are used as nodes, and differential units and reference changes are used as edge relationships; the author, timestamp, source system, session identifier, and editing intent tag are recorded on the edges to form a version causal relationship network; edges point from the previous version to the next version, with differential units as edge attributes; reference changes are another type of edge, forming a causal network structure of the version chain. An archived evidence package is generated for each version. The evidence package supports location and verification by version number, author, time window, and referenced object. During the verification process, the fingerprint in the evidence package is compared with the fingerprint tree in the historical archive database to verify the validity, completeness, and consistency of the document.

[0018] Further, the specific process for performing consistency verification and outputting the deduplicated archived results of cloud documents is as follows: The completeness of the evidence elements and the consistency verification results are statistically analyzed; the text snapshot, differential summary, object list, citation relationships, permission snapshot, and session summary elements already present in the evidence package are counted and compared with the evidence element list that the version should have, and the size of the intersection is calculated as the number of existing elements; the size of the set of elements that should be present is calculated as the number of elements that should be present; the element completeness is obtained by dividing the number of existing elements by the number of elements that should be present, and the type of missing elements, corresponding index key, and missing reason code are written... The version verification record is entered; at the same time, consistency verification is performed on the text fingerprint, object fingerprint and reference fingerprint, respectively, and they are compared with the corresponding fingerprints stored in the evidence package and the fingerprint index registered in the historical archive. When all three types of fingerprints are equal, the consistency verification is deemed to have passed; when any fingerprint is inconsistent, the consistency verification is deemed to have failed, and failure handling is performed: the version is marked as to be rebuilt, the archive effective mark is frozen, the inconsistent fingerprint type and the source of difference are recorded, and the process entry of supplementary collection, re-parsing or rollback and rewriting of evidence package is triggered, and the consistency verification conclusion and cloud document deduplication archive result are output.

[0019] Furthermore, the second aspect of the present invention provides a cloud document deduplication archiving and traceability management system for office knowledge bases, applied to a cloud document deduplication archiving and traceability management method for office knowledge bases, comprising: a multi-source document acquisition module, used for event-driven acquisition of document objects, obtaining the original dataset and performing structured parsing to obtain a full data stream, and performing normalization processing and anomaly marking on the full data stream; a multi-layer fingerprint generation module, used for generating multi-layer fingerprints based on the full data stream, performing candidate duplicate retrieval and alignment, and performing archiving duplicate evaluation, filtering duplicate candidate documents, and establishing a version chain entry point; a differential redundancy module, used for generating differential units based on the version chain entry point and performing intent strength evaluation, storing differential unit data and constructing an indexing strategy; and a knowledge base retrieval service module, used for constructing a version causal relationship network, generating an archiving evidence package, performing consistency verification, and outputting the cloud document deduplication archiving results.

[0020] Beneficial effects

[0021] The present invention has the following beneficial effects:

[0022] (1) This invention generates standardized document frames by performing event-driven collection and structured parsing of document objects from multiple sources, unifying field definitions and record granularity, thereby improving cross-platform data access consistency and processing availability. It reduces interference from missing text, parsing failures, insufficient permissions, and corrupted objects on the deduplication and tracing process, thus improving processing stability.

[0023] (2) This invention, by constructing a multi-layer fingerprint system, can distinguish between documents with similar content but different structures, and documents with similar structures but different dependencies, thereby reducing false positives and false negatives. Using content fingerprints and block-level fingerprints as primary keys, candidate documents are retrieved from the historical archive and block-level alignment is performed, outputting block overlap ratio, structural consistency, and differential summaries, thus improving candidate recall efficiency and reducing the overhead of full-database comparison.

[0024] (3) In this invention, by measuring the consistency of content similarity, block-level similarity, structural similarity and dependency similarity and using a fusion method of multiplication and square root to form the archive duplication evaluation value, the difference at any level can significantly lower the comprehensive score, thereby suppressing the erroneous merging caused by high similarity of a single feature, reducing archive redundancy and maintaining the continuity of the version chain.

[0025] (4) This invention generates version difference units and constructs editing sessions and intent tags by performing block-level differential on adjacent versions of the same main document version chain, thereby enabling interpretable annotation of supplementary explanations, structural adjustments, object replacements, reference changes, and permission changes, and enhancing auditing and review capabilities. It supports one-click playback, comparison, auditing, and recovery of versions, improving the reliability and verifiability of version reproduction.

[0026] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0027] Figure 1 This is a flowchart of the cloud-based document deduplication, archiving, and traceability management method for office knowledge bases according to the present invention.

[0028] Figure 2 This is an architecture diagram of the cloud-based document deduplication, archiving, and traceability management system for office knowledge bases according to the present invention.

[0029] Figure 3 This is a graph showing the results of the document archiving duplication assessment for this invention.

[0030] Figure 4 This invention provides a temporal variation diagram of the intensity of the editing session operation intent.

[0031] Figure 5 This is a diagram illustrating the completeness of evidentiary elements and the credibility of traceability in this invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Please see Figures 1-5 This invention provides a technical solution: a cloud document deduplication archiving and traceability management method for office knowledge bases, comprising: S1, performing event-driven acquisition of document objects, obtaining the original dataset and performing structured parsing to obtain a full data stream, and performing normalization processing and anomaly marking on the full data stream; S2, generating multi-layer fingerprints based on the full data stream, performing candidate duplicate retrieval and alignment, and performing archiving duplicate evaluation, filtering duplicate candidate documents, and establishing a version chain entry point; S3, generating differential units based on the version chain entry point and performing intent strength evaluation, storing differential unit data and constructing an indexing strategy; S4, constructing a version causal relationship network, generating an archiving evidence package, performing consistency verification, and outputting the cloud document deduplication archiving results.

[0034] Specifically, the process involves event-driven data collection of document objects to obtain the raw dataset and perform structured parsing to obtain the full data stream. The normalization and anomaly marking of the full data stream are performed as follows: Event-driven data collection is conducted on document objects from the office knowledge base, originating from cloud document collaboration platforms, cloud drive directories, email attachments, instant messaging files, and exported meeting minutes, to obtain the raw dataset. The raw dataset includes text content, heading levels, table objects, image objects, attachment references, comments and annotations, and permission sharing status. Simultaneously, the collection timestamp, source system identifier, document unique identifier, and version number are recorded. The index view of the historical archive is synchronously accessed as a comparison benchmark input. The historical archive stores the main document record, version chain record, candidate documents, differential records, and multi-level fingerprint index of archived documents. The historical archive includes the main document identifier, version number, fingerprint tree, content fingerprint, block-level fingerprint, structural fingerprint, dependency fingerprint, and a list of associated objects and their reference relationships.

[0035] The system performs structured parsing and unified encapsulation on the original dataset to generate a full data stream. The text content is segmented into paragraph blocks, and block-level position indexes are preserved. An object list is generated for table objects, image objects, collection timestamps, source system identifiers, and document unique identifiers, and reference relationships are established. Normalization processing using a caliber dictionary mapping is performed on the full data stream. The caliber dictionary includes field naming conventions (e.g., unified field names for creation time, update time, and author fields from different sources), encoding conventions (time fields are unified to the same time zone and time representation; author and organization path are unified to standardized encoding), structured field extraction rules (e.g., priority and fallback rules for extracting author, path, and version number from email attachment headers, cloud storage metadata, and collaboration platform attributes), and value domain normalization rules (e.g., null values, placeholders, illegal character cleaning, and truncation strategies). Mapping, validation, and write-back are performed field-by-field on each document frame, and the matching mapping rule number and mapping confidence flag are recorded.

[0036] For full data streams with missing text, parsing failures, insufficient permissions, or corrupted objects, anomaly markers are written and the data is isolated. Standardized document frames, after normalization and anomaly marking, are written to the archive buffer and simultaneously bound to the candidate master document entry key in the historical archive. The archive buffer is a combination of a document frame writing queue and an index table. The queue temporarily stores standardized document frames that have passed normalization and quality checks in the order of acquisition, while the index table quickly locates the document frame to be processed using the entry key. The writing logic is as follows: when a document frame does not meet the anomaly isolation condition, it is written to the queue, and the entry key, unique document identifier, version number, source system identifier, and queue offset position are written to the index table. The candidate master document entry key is generated by concatenating the source system identifier and the document unique identifier, and is used to locate the cross-version chain of the same document within the same source system. When the source system does not guarantee the stability of the document unique identifier, the entry key is generated by combining the source system identifier, the normalized organizational path, the author identifier, and the fingerprint of the text content, so that the candidate master document can still be located in the scenario of cross-directory migration or renaming. The entry key is written into the index table of the pending archive buffer and serves as the primary search key for retrieving candidate master documents in the historical archive. After a match, the master document identifier and version chain entry are returned.

[0037] Anomaly marking and isolation employs an anomaly queue isolation method: when missing text, parsing failure, insufficient permissions, or object corruption are detected, an anomaly type flag, trigger field, and reason code are written to the document frame, and the document frame is written to the anomaly queue; document frames in the anomaly queue are stopped from entering subsequent deduplication and archiving processes, or enter the review processing channel with low priority, in order to avoid abnormal data participating in fingerprint generation and duplication assessment, which could lead to erroneous merging or deduplication.

[0038] In this implementation scheme, by using event-driven acquisition and structured parsing, document objects from multiple sources are encapsulated into a computable full data stream under the same field scope and record granularity. The consistency of key meta-information such as time, author, path, and version is achieved by mapping with a scope dictionary. This ensures that documents across platforms, directories, and formats have a unified alignment basis before entering the subsequent fingerprint generation and deduplication evaluation. It avoids amplifying fingerprint bias and misjudgment of repetition by abnormal samples, improves the stability of candidate retrieval hits, and reduces the probability of misclassification and omission.

[0039] Specifically, the process of generating multi-layer fingerprints based on the full data stream and performing candidate duplicate retrieval and alignment is as follows: First, multi-layer fingerprints are generated based on the full data stream. Then, based on the text content field in the full data stream, text normalization is performed, including uniform encoding to UTF-8, uniform full-width and half-width characters, uniform uppercase and lowercase, continuous whitespace folding, removal of invisible control characters, trimming of trailing spaces, uniform newline characters to "\n", uniform table transcribing placeholders, uniform image and attachment placeholders, and synonym replacement for numbered lists and bullet points. Simultaneously, comment annotations are extracted from the text string, retaining only the main body of the text, to avoid instability in the content fingerprint due to collaborative noise.

[0040] Next, the normalized text string is subjected to cryptographic hashing to obtain the content fingerprint. Based on the set of text blocks segmented by paragraph blocks in the full data stream and the block-level position index, the normalized line break boundary is used as the basic segmentation point, and the text is further segmented by combining the heading level and style block boundary. SHA-256 cryptographic hashing is performed on each paragraph block to obtain the paragraph block fingerprint. The paragraph block fingerprints are sorted by position index to form a sequence, and Merkle tree aggregation is used to obtain the block-level fingerprint. The leaf nodes of the Merkle tree are the paragraph block fingerprints sorted by position index, and the parent node is obtained by concatenating the fingerprints of two adjacent child nodes in a fixed order and then performing SM3 cryptographic hashing. Based on the heading level, directory structure, and object list in the full data stream, structured serialization is performed using deterministic serialization rules. The heading hierarchy is expanded into an ordered sequence of nodes based on hierarchy number, heading text, and parent-child relationships; the directory structure is expanded into a sequence of directory path fragments; the object list is expanded into an object list based on object type, object identifier, object occurrence number, and object summary; the three types of sequences are concatenated in a fixed field order, and all fields are escaped and delimited to obtain a structure serialization string. The string is then subjected to SM3 cryptographic hashing to obtain a structure fingerprint, ensuring that the same structure on different platforms yields a consistent structure fingerprint after serialization. Based on the reference relationships and attachment list established in the full data stream, the reference edges are normalized and sorted according to the priority of the referenced object type, the lexicographical order of the referenced object identifier, and the ascending order of the occurrence number. The dependency fingerprint is used as a clear distinguishing point, and cryptographic hashing is performed to obtain the dependency fingerprint.

[0041] Even when the fingerprints of the main content are consistent, there is still a risk of erroneous merging of documents with the same main content but different external dependencies (e.g., referencing different attachment versions, different cloud drive links, different meeting materials, or different shared objects with different permissions). By generating dependency fingerprints from the citation relationships and attachment lists, and using them as independent evidence input in duplication assessment and merging decisions, documents with inconsistent dependencies can be separated from the same version chain or an archiving strategy that only differentially enters the chain but retains dependency changes can be triggered. This reduces the probability of erroneous merging (misclassifying documents with different dependencies as the same version) and omissions (failing to identify them as the same document due to structural differences), and improves the reproducibility of attachments, links, and referenced objects during retrospective playback.

[0042] Candidate duplication retrieval and alignment are performed within multi-level fingerprints. Candidate documents in the historical archive are retrieved using content fingerprints and block-level fingerprints as primary keys. Block-level alignment is performed between candidate documents and the current document. The content fingerprint and block-level fingerprint of the current document to be archived are used as search keys in the archive buffer. First, a set of candidate main documents with completely identical content fingerprints is retrieved from the content fingerprint index of the historical archive. Then, a set of candidate documents with identical block-level fingerprints is retrieved from the block-level fingerprint Merkle root index. The two sets of candidate documents are then deduplicated and merged to obtain a candidate document list. Subsequently, for each candidate document, its paragraph block fingerprint sequence is compared with the paragraph block fingerprint of the current document. The sequence is aligned at the block level: using paragraph block fingerprint equivalence matching as anchor points, the common anchor point set of the two sequences is first located; then, the text is divided into several alignment intervals according to the relative order of the anchor points in the sequence; within each interval, the number of matching blocks, the number of unmatched blocks, and the number of suspected rearranged blocks are calculated, and the block overlap ratio is obtained accordingly; at the same time, structural consistency verification is performed on the heading level sequence and the object list sequence, and the structural consistency results are output; a duplicate candidate set is output, and a differential summary is generated for each candidate document. The differential summary includes: a list of newly added blocks, a list of deleted blocks, a list of replaced blocks, rearranged block pairs, a list of object reference changes, and a list of dependency changes.

[0043] In this implementation plan, deterministic normalization, block segmentation, and multi-layer fingerprint construction are performed on the full data stream to ensure that the same document can still obtain stable and consistent content fingerprints, block-level fingerprints, structural fingerprints, and dependency fingerprints under cross-platform, cross-format, and cross-collaboration noise conditions. This reduces misclassification and omissions caused by reordering, renaming, migration, attachment replacement, or link changes, improves the interpretability and auditability of deduplication archiving decisions, and provides a reliable basis for the reproducibility of version linking, dependency change tracking, and replay.

[0044] Specifically, the process of evaluating the archiving duplication is as follows: the similarity between the current document to be archived and the candidate document at four levels, namely content fingerprint, block fingerprint, structural fingerprint and dependency fingerprint, is calculated by using the consistency measurement function; the uniform output range of the consistency measurement function is [0,1], and the larger the value, the more consistent it is; (1) Content similarity: the content fingerprints of the two documents are compared equally; when the content fingerprints of the two documents are consistent, the content similarity is 1, and when the content fingerprints of the two documents are inconsistent, the content similarity is 0, which is used to express the strong consistency judgment of whether the full text content is completely consistent.

[0045] (2) Block-level similarity: Calculate the overlap of the paragraph block fingerprint sets of the two documents; find the intersection of the paragraph block fingerprint sets of the two documents to obtain the number of common blocks; find the union of the paragraph block fingerprint sets of the two documents to obtain the total number of blocks; divide the number of common blocks by the total number of blocks to obtain the block-level similarity, which is used to characterize whether the content blocks are highly overlapping and whether there are additions, deletions, modifications and rearrangements.

[0046] (3) Structural similarity: The structural serialization results of the two documents are compared for consistency; the title level sequence and the object list sequence are converted into ordered token sequences respectively, the number of matching tokens with the same position or the same path in the two sequences is counted, and the number of matching tokens is divided by the average of the total number of tokens in the two sequences to obtain the structural similarity, which is used to characterize whether the title organization, directory path and object layout are consistent.

[0047] (4) Dependency similarity: Calculate the set overlap of the reference relationship between the two documents and the dependency token set generated by the attachment list; the dependency token must contain at least a triplet of reference object type, reference object identifier and occurrence number. Find the intersection and union of the dependency token sets of the two documents and obtain the dependency similarity by the intersection-union ratio, which is used to characterize the difference between the same text but different external dependencies.

[0048] The archive duplication evaluation value is obtained by multiplying the content similarity, block-level similarity, structural similarity and dependency similarity together and then taking the fourth power. Then, deduplication and archiving are performed.

[0049] The specific formula for calculating the archived redundancy assessment value is as follows:

[0050] ;

[0051] In the formula, This represents the archive duplication assessment value, used to quantify the overall degree of duplication between the current document to be archived and historical candidate documents; This represents a content fingerprint, used as a primary key matching item during the candidate retrieval stage; This represents a block-level fingerprint, used to calculate the degree of block overlap. Structural fingerprints are used to characterize differences between similar contents but with different structural organization. Dependency fingerprints are used to characterize differences where the main content is the same but the dependent objects are different. This represents the document currently to be archived, provided through the full data stream and obtained when entering the deduplication process, and used as the write object for deduplication archiving; Candidate documents are retrieved by searching for candidate main documents or candidate versions using content fingerprints and block-level fingerprints, and are used as a reference for the current documents to be archived; The consistency metric function is obtained by calculating the similarity between two documents on the fingerprint at this layer, and is used as a uniform dimensional input for the four types of duplicate evidence.

[0052] When a dimension cannot be computed due to missing fields, parsing failure, or empty dependencies, it is not directly set to 0; instead, the dimension is marked as a missing dimension and removed from the geometric convergence. The evaluation value is obtained by multiplying the similarities of the remaining available dimensions and taking the square root of the corresponding number of dimensions. When the number of available dimensions is 0, the evaluation value is set to 0 and an unevaluable reason code is output, causing the archiving process to be transferred to manual review or re-collection and re-parsing queues. When both documents have empty dependencies and the dependency fields are valid (not parsing failure), the dependency similarity is set to 1; when only one document has an empty dependency and the other does not, the dependency similarity is set to 0, to explicitly distinguish between no dependencies and missing or inconsistent dependencies.

[0053] Table 1 shows the archive duplication assessment data, used to evaluate the degree of archive duplication between the current document and the candidate main document from multiple dimensions of similarity indicators. DOC-001: Candidate main document ID is MAIN-005, content similarity is 0.92, block-level similarity is 0.88, structural similarity is 0.9, dependency similarity is 0.85, and archive duplication assessment value is 0.89; DOC-002: Candidate main document ID is MAIN-008, content similarity is 0.85, block-level similarity is 0.82, structural similarity is 0.78, dependency similarity is 0.8, and archive duplication assessment value is 0.81; DOC-003: Candidate main document ID is MAIN-003, content similarity is 0.72, block-level similarity is 0.68, structural similarity is 0.7, dependency similarity is 0.7, and dependency similarity is 0.85. The similarity score is 0.65, and the archive duplication assessment value is 0.69; DOC-006: Candidate main document ID is MAIN-005, content similarity is 0.95, block-level similarity is 0.93, structural similarity is 0.91, dependency similarity is 0.89, and archive duplication assessment value is 0.92; DOC-007: Candidate main document ID is MAIN-012, content similarity is 0.78, block-level similarity is 0.75, structural similarity is 0.73, dependency similarity is 0.76, and archive duplication assessment value is 0.75; DOC-008: Candidate main document ID is MAIN-009, content similarity is 0.83, block-level similarity is 0.8, structural similarity is 0.82, dependency similarity is 0.79, and archive duplication assessment value is 0.81.

[0054] Table 1 Archived Repeatability Evaluation Data Table

[0055] Current document ID Candidate main document ID Content similarity Block-level similarity Structural similarity Dependence on similarity Archived repeatability evaluation value DOC-001 MAIN-005 0.92 0.88 0.9 0.85 0.89 DOC-002 MAIN-008 0.85 0.82 0.78 0.8 0.81 DOC-003 MAIN-003 0.72 0.68 0.7 0.65 0.69 DOC-006 MAIN-005 0.95 0.93 0.91 0.89 0.92 DOC-007 MAIN-012 0.78 0.75 0.73 0.76 0.75 DOC-008 MAIN-009 0.83 0.8 0.82 0.79 0.81

[0056] like Figure 3 The document archiving duplication assessment result graph shown below uses the horizontal axis to represent the current document number to be archived and the vertical axis to represent the archive duplication assessment value. The dashed line represents the deduplication threshold, used to determine whether to merge the document into the main document version chain or to add it as a new main document. Green bars in the graph represent documents with assessment values ​​not less than the deduplication threshold; the system determines these documents as mergeable and executes the merge decision. Red bars represent documents with assessment values ​​less than the deduplication threshold; these are determined as non-duplicate main documents and a new document creation decision is executed. This demonstrates how a unified mechanism of archive duplication assessment value and threshold judgment enables interpretable deduplication and merging of different documents, and separates the flow of new document entry.

[0057] In this implementation plan, redundant storage is avoided and the consistency of the version chain is ensured through a comprehensive evaluation based on multi-dimensional similarity. For documents with evaluation values ​​less than the deduplication threshold, they will be entered into the database as new master documents, and a new version chain entry will be established to ensure the uniqueness and accuracy of documents. This can effectively distinguish between duplicate documents and new documents, and at the same time realize the intelligent diversion of deduplication and new entry into the database, thereby greatly improving archiving efficiency and reducing storage redundancy.

[0058] Specifically, the process of filtering duplicate candidate documents and establishing version chain entry points is as follows: The archive duplication evaluation value and deduplication threshold are compared in real time. When the archive duplication evaluation value is not less than the deduplication threshold, the current document is merged into the main document version chain. The main document record is located using the candidate main document entry key. The main document version chain tail pointer is locked, and the previous version number is read and written into the version chain incremental record set. Evidence package entries and causal network edge records corresponding to the version are generated. The evidence package entries include text snapshot fingerprints, block-level fingerprint roots, structural fingerprints, dependency fingerprints, object list fingerprints, reference relationship fingerprints, and permission snapshot fingerprints. Differential units and reference and permission changes are written to the archive incremental area without repeatedly writing the complete text. The main document version chain tail pointer is updated, and this merging is committed. The process involves transactions where, if writing any sub-item fails, the version chain tail pointer and edge records are rolled back to avoid redundancy while ensuring the version chain and causal network remain unbroken. When the archive duplication assessment value is less than the deduplication threshold, the current document is added to the database as the new master document, and a version chain entry is established. Simultaneously, the master document evidence package and the causal network root node are initialized, and the first version's text snapshot or text compressed storage pointer, object list, reference relationship, permission snapshot, and its corresponding fingerprint are written for playback and verification. The causal network root node is initialized synchronously, and node attributes (version number, fingerprint root, object list index, reference index, permission index) are written with the first version as the root node. The root node and the first version are used as the starting association record. The new master document entry key is written back to the historical archive database index view.

[0059] In this implementation plan, through real-time judgment and transactional writing mechanism of archived duplication evaluation value and deduplication threshold, duplicate documents are automatically merged into the existing version chain or new documents are automatically added to the database: while avoiding duplicate storage of complete text, it ensures that the version chain and causal network edge records are continuous and traceable; and ensures that retrieval, playback, verification and auditing all have a consistent version chain starting point and verifiable evidence support.

[0060] Specifically, the process of generating differential units based on the version chain entry and evaluating intent strength is as follows: For two adjacent versions merged into the same main document version chain, the block-level alignment result and object reference relationship are called. Block-level differential is performed on the text paragraph block and the object list. New blocks, deleted blocks, replaced blocks, rearranged blocks, differential types, and object reference change types are identified to generate differential units. Each differential unit is bound to a block-level position index and a source version number. Operation intent labels are constructed based on differential units: continuous differential units are aggregated into editing sessions according to the same author, the same time window, and the same location neighborhood. Editing intent labels are output according to differential type within the editing session. Editing intent labels include supplementary explanations, structural adjustments, object replacements, reference changes, and permission changes.

[0061] The operation type weights corresponding to the differential units are obtained through an evidence scoring method. The differential type is used as the primary evidence, and changes in the object list, object reference type, and permission field are used as secondary evidence. The operation type weights are obtained by normalizing the evidence scores corresponding to each intent tag. The change in structural depth is obtained by comparing the absolute value of the change in the hierarchical depth of the document outline before and after the editing session. The maximum structural depth is obtained by analyzing the title hierarchy structure tree of the document version before editing. The number of external references is obtained by counting the number of differential units in this editing session that involve the addition, deletion, or modification of external reference content. The number of permission changes is obtained by counting the number of differential units in this editing session that involve changes to the document. The differential unit index is obtained by traversing all differential units contained in this editing session.

[0062] For each differential unit within an editing session, the operation type weight of the differential unit is multiplied by the structural depth influence coefficient to obtain the basic contribution value. The structural depth change is incremented by one and then logarithmized. The logarithm is then divided by the maximum structural depth of the document, incremented by one, and then logarithmized again. One is added to the ratio to obtain the adjustment factor. The basic contribution value is multiplied by the adjustment factor and then summed sequentially to obtain the total contribution score. The citation change coefficient is multiplied by the number of external citations to obtain the citation bonus, and the permission change coefficient is multiplied by the number of permission changes to obtain the permission bonus. The total contribution score is added to the citation bonus and the permission bonus to obtain the intent intensity assessment value.

[0063] The specific formula for calculating the intent intensity assessment value is as follows:

[0064] ;

[0065] In the formula, The intent strength assessment value is used to comprehensively quantify the semantic importance and structural impact of a single editing session; This represents the operation type weight corresponding to the i-th difference unit, with a value range of [1,5], used to distinguish the inherent importance of different operation types; The depth influence coefficient of the block affected by the differential unit is obtained by the header level structure tree positioning and normalized depth scoring method. The value range is (0,1], which is used to reflect the difference in the impact of the modification occurring at different levels of the document. This indicates the amount of change in structural depth, used to quantify the extent to which this edit has altered the overall structural depth of the document. This indicates the maximum structural depth, used to relative evaluate changes in structural depth within the context of the entire document; The reference change coefficient is obtained by performing a reference edge difference scoring method on the reference relationship between the start version and the end version of the same editing session. The value range is (0,5], and it is used to significantly improve editing sessions that involve external reference changes. This indicates the number of external references, used to measure the extent to which editing breaks or updates external dependencies. The permission change coefficient is obtained by performing a permission field differential scoring method on the permission sharing status of the start version and the end version of the session within the same editing session. The value range is (0,5], and it is used to significantly improve editing sessions that involve changes in document access permissions, sharing settings security, or collaboration attributes. This indicates the number of permission changes, used to measure the extent to which editors alter document security and collaboration modes; This represents the difference cell index, used to uniquely identify and traverse each difference cell within the computation session in the formula.

[0066] like Figure 4 The graph shows the temporal variation of the intensity of editing session operation intent. The horizontal axis represents the editing session number, and the vertical axis represents the intensity assessment value of the operation intent corresponding to that session. The red dashed line in the graph represents the intent threshold, used to distinguish the storage and indexing strategies for sessions with different intensities. The blue broken line and dots represent the change of the intent intensity of each session with the session sequence: when the intent intensity assessment value of a session is not less than the intent threshold, the session is marked as full storage, meaning that the original text of the content block involved and the snapshot of the embedded object are retained in the differential unit, and the session is written as a high-weight index into the version chain and traceability graph; when the intent intensity assessment value of a session is less than the intent threshold, the session is marked as fingerprint storage, meaning that only the corresponding content fingerprint and differential summary index are retained to reduce storage overhead and prevent low-value details from entering the high-priority traceability link. This graph intuitively illustrates the decision boundary of the intent intensity threshold for session-level traffic splitting, and the dynamic switching effect of the intent intensity fluctuation of different editing sessions on the same document version chain on storage strategies.

[0067] In this implementation plan, by uniformly quantifying block-level differential results, structural depth changes, external reference changes, and permission sharing changes into session-level intent strength, automatic classification and interpretable annotation of editing sessions are achieved. This allows high-impact sessions to be fully recorded first, while low-impact sessions are stored in the database in a lightweight manner using fingerprints and summaries. This reduces redundant differential storage while maintaining the retrieval, verification, and replayability of key semantic changes and security collaboration changes in the version chain.

[0068] Specifically, the process of storing differential cell data and constructing an index strategy is as follows: For editing sessions with an intent intensity assessment value greater than the intent threshold, a snapshot of the original text of the content block and embedded objects is fully preserved in the differential cell; for editing sessions with an intent intensity assessment value less than or equal to the intent threshold, only the content fingerprint is preserved to optimize storage; the intent intensity assessment value is written as a dynamic weight factor into the document field weight of the inverted index. This allows the results to be sorted and filtered based on this weight when responding to user requests for version retrieval, key node playback, or change tracking, thereby prioritizing and covering version change events with stronger intent and more critical impact.

[0069] This implementation plan optimizes storage efficiency and reduces redundant data usage. When responding to user requests for version retrieval, key node playback, or change context tracing, it can dynamically sort and filter results based on intent intensity, prioritizing version change events with stronger intent and more critical impact. This significantly improves retrieval accuracy, playback relevance, and the clarity of context tracing, enhancing the overall user experience.

[0070] Specifically, the process of constructing a version causal relationship network and generating archived evidence packages is as follows: A traceable causal graph is constructed based on version chain entry points, differential units, and operation intent annotations. Versions are used as nodes, and differential units and reference changes are used as edge relationships. Authors, timestamps, source systems, session identifiers, and editing intent tags are recorded on the edges to form a version causal relationship network. Edges point from the previous version to the next version, with differential units as edge attributes. Reference changes are another type of edge, forming a causal network structure of the version chain. An archived evidence package is generated for each version, and the version text snapshot fingerprint, block-level differential digest, object list, reference relationships, permission snapshot, operation session digest, and verification hash are written into the evidence package index. The evidence package supports location and verification by version number, author, time window, and referenced object. During verification, the fingerprint in the evidence package is compared with the fingerprint tree in the historical archive to verify the validity, completeness, and consistency of the document.

[0071] This implementation plan transforms complex version histories into structured causal networks, intuitively presenting the complete path and dependencies of document evolution, making the origins and consequences of each change readily apparent. By archiving evidence packages, the complete operational context and content fingerprint of each version are solidified, forming an immutable and auditable record, providing a credible basis for evidence preservation, auditing, and accountability.

[0072] Specifically, the process of performing consistency verification and outputting the deduplicated cloud document archiving results is as follows: The completeness of evidence elements and the consistency verification results are statistically analyzed; the text snapshot, differential digest, object list, citation relationships, permission snapshot, and session digest elements already present in the evidence package are counted and compared with the evidence element list that the version should have, and the size of the intersection is calculated as the number of existing elements; the size of the set of elements that should be present is calculated as the number of elements that should be present; the element completeness is obtained by dividing the number of existing elements by the number of elements that should be present, and the type of missing elements, corresponding index key, and missing reason code are written into the version verification record; simultaneously, consistency verification is performed on the text fingerprint, object fingerprint, and citation fingerprint, comparing them with the corresponding fingerprints stored in the evidence package and the fingerprint indexes registered in the historical archive. Perform an equivalence comparison; when all three fingerprint types are identical, the consistency check is considered successful; when any fingerprint is inconsistent, the consistency check is considered unsuccessful, and failure handling is performed: mark the version as pending reconstruction, freeze the archive effective flag, record the inconsistent fingerprint type and source of difference, and trigger the process entry point for supplementary collection, re-analysis, or rollback and rewriting of the evidence package, output the consistency check conclusion, calculate the traceability credibility score based on the element completeness and the consistency check conclusion, and compare it with the credibility threshold: when the traceability credibility score is not less than the credibility threshold, mark the version as a traceable version and allow one-click playback; when the traceability credibility score is less than the credibility threshold, mark the version as pending completion, and output the missing element list, missing reason code, and suggested completion path for supplementary collection, re-analysis, or reconstruction of evidence.

[0073] Outputs deduplicated cloud document archiving results, providing each document with its main document number, version number, deduplication archiving decision result, editing intent tag, and evidence package index. It offers a search entry point based on the main document number and version number, supporting location by time, author, source system, object type, and fingerprint conditions. It provides a one-click version playback service, restoring a specified version view based on the text snapshot, object list, citation relationships, and permission snapshot in the evidence package. It offers a version comparison service, displaying differential summaries and object citation changes between adjacent versions and any two versions. It provides an audit service, outputting audit records based on session summaries, permission changes, and citation changes. Finally, it provides a recovery service, supporting the export or re-import of specified versions to the target collaboration platform, enabling the location, reproduction, and traceability of any document version.

[0074] like Figure 5The diagram showing the completeness of evidence elements and traceability credibility analysis illustrates this. The upper part of the diagram shows the changing trends of element completeness and consistency verification values ​​for each version as the version evolves, with the red dashed line representing the credibility threshold. For example, V1.0 has an element completeness of 0.83 and a consistency verification value of 0.90; V1.1 has 0.92 and 0.95 respectively; V1.2 decreases to 0.75 and 0.70 respectively; V1.3 has 0.95 and 0.98 respectively; V2.0 has 0.68 and 0.65 respectively; and V2.1 has 0.98 and 0.99 respectively. The lower part of the diagram shows the traceability credibility score and judgment results formed by the above completeness and consistency, with the red vertical dashed line also representing the threshold of 0.8: when the score is not less than 0.8, it is marked as a "traceable version"; when the score is less than 0.8, it is marked as needing to be supplemented. In the figure, V1.0 (0.86), V1.1 (0.93), V1.3 (0.96) and V2.1 (0.98) all passed the threshold and were determined to be traceable versions; V1.2 (0.73) and V2.0 (0.67) were below the threshold and were determined to be versions that need to be supplemented, thus clearly indicating the version nodes on the version chain that need to supplement, re-parse or reconstruct evidence.

[0075] This implementation plan achieves fine-grained hierarchical management of the version control repository. Any inconsistency triggers automated failure handling and repair processes, thereby fundamentally maintaining the integrity and reliability of the archive repository. It significantly improves operational efficiency, enabling rapid location and repair of problematic nodes in the data chain. It allows for secure and convenient version identification, difference analysis, operation auditing, and content recovery, fully empowering document traceability, review, and reuse.

[0076] Specifically, the second aspect of this invention provides a cloud-based document deduplication, archiving, and traceability management system for office knowledge bases, applied to a cloud-based document deduplication, archiving, and traceability management method for office knowledge bases. The system includes: a multi-source document acquisition module for event-driven acquisition of document objects, including subscription-triggered acquisition of events such as creation, editing and saving, moving and renaming, permission changes, comment annotations, attachment additions / deletions, and link updates; obtaining the original dataset and performing structured parsing to obtain a full data stream; normalizing and anomaly marking the full data stream; and a multi-layer fingerprint generation module for generating multi-layer fingerprints based on the full data stream, performing candidate duplicate retrieval and alignment, evaluating archiving duplication, filtering duplicate candidate documents, and establishing a version chain entry point. The filtering and entry point establishment include real-time comparison of the archiving duplication evaluation value and the deduplication threshold; when the threshold condition is met, the document is deduplicated. The current document is written to the version chain incremental record set and corresponding evidence package entries and network edge records are generated. Conversely, a new main document record, version chain entry, and root evidence package are initialized to ensure that the version chain and tracing network remain unbroken. The differential redundancy module is used to generate differential units based on the version chain entry and perform intent strength evaluation. It stores the differential unit data and builds an indexing strategy. When the intent strength is less than the intent threshold, only the content fingerprint and differential summary are retained to reduce storage redundancy. At the same time, the intent strength is written as an index enhancement factor into the retrieval index field and version graph edge attributes. The knowledge base retrieval service module is used to build a version causal relationship network, generate archived evidence packages, and perform consistency verification. Consistency verification includes comparing the text fingerprint, object list fingerprint, and reference fingerprint stored in the evidence package with the fingerprint tree root of the corresponding version in the historical archive and outputting the cloud document deduplication archive results.

[0077] This implementation plan achieves accurate and efficient duplicate identification and automatic version lineage construction, automatically establishing a clear and accurate document version evolution path, laying the foundation for traceability management. It reliably provides a one-stop service for version playback, comparison, auditing, and recovery, meeting various needs for document location, reproduction, analysis, and auditing.

[0078] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0079] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A cloud-based document deduplication, archiving, and traceability management method for office knowledge bases, characterized in that: Includes the following steps: S1: Perform event-driven data collection on the document object, obtain the raw dataset and perform structured parsing to obtain the full data stream, and perform normalization and anomaly marking on the full data stream; S2 generates multi-layer fingerprints based on the full data stream, performs candidate duplication retrieval and alignment, evaluates archive duplication, filters duplicate candidate documents, and establishes a version chain entry point. S3 generates differential units based on the version chain entry and performs intent strength assessment, stores the differential unit data and builds an indexing strategy; S4 constructs a version causal relationship network, generates an archived evidence package, performs consistency verification, and outputs the deduplicated archived results of cloud documents.

2. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process of performing event-driven data collection on document objects, obtaining the raw dataset, performing structured parsing to obtain the full data stream, and normalizing and anomaly marking the full data stream is as follows: Event-driven data collection is performed on document objects from cloud document collaboration platforms, cloud drive directories, email attachments, instant messaging files, and exported meeting minutes in the office knowledge base to obtain the raw dataset. The raw dataset includes text content, heading levels, table objects, image objects, attachment references, comments and annotations, and permission sharing status. The collection timestamp, source system identifier, document unique identifier, and version number are recorded simultaneously. The index view of the historical archive is also accessed synchronously as the comparison benchmark input. The historical archive is used to store the main document record, version chain record, candidate document, differential record, and multi-level fingerprint index of archived documents. The original dataset is subjected to structured parsing and unified encapsulation to generate a full data stream. Normalization processing of the full data stream is performed using a caliber dictionary mapping. Write exception flags to the full data stream containing missing text, parsing failures, insufficient permissions, and corrupted objects, and isolate them; Normalized document frames, which are normalized and marked for anomalies, are written to the archive buffer and bound to the entry key of the candidate master document in the historical archive.

3. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process of generating multi-layer fingerprints based on the full data stream and performing candidate duplicate retrieval and alignment is as follows: Multi-layer fingerprints are generated based on the full data stream. First, text normalization is performed on the main text content field of the full data stream, followed by encrypted hashing of the normalized text string to obtain the content fingerprint. Then, based on the set of main text blocks segmented by paragraph blocks and the block-level position index in the full data stream, the normalized line break boundaries are used as the basic segmentation points, and the main text is further segmented by combining the heading level and style segment boundaries. Each paragraph block is then encrypted and hashed using SHA-256 to obtain the paragraph block fingerprint. The paragraph block fingerprints are sorted by position index to form a sequence, and Merkle tree aggregation is used to obtain the block-level fingerprint. The leaf nodes of the Merkle tree are the paragraph block fingerprints sorted by position index, and the parent node is obtained by concatenating the fingerprints of two adjacent child nodes in a fixed order and then performing SM3 encrypted hashing. Finally, based on the heading level, directory structure, and object list in the full data stream, structured serialization is performed, and encrypted hashing is executed to obtain the structure fingerprint. Based on the reference relationships and attachment list established in the full data stream, the reference edges are normalized and sorted by priority of reference object type, lexicographical order of reference object identifier and ascending order of occurrence number. The dependency fingerprint is used as a clear distinguishing point, and cryptographic hashing is performed to obtain the dependency fingerprint. Perform candidate duplication retrieval and alignment within multi-layer fingerprints; retrieve candidate documents from the historical archive using content fingerprints and block-level fingerprints as primary keys; perform block-level alignment between candidate documents and the current document to obtain block overlap ratio and structural consistency, and output a duplicate candidate set.

4. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process for conducting archive duplication assessment is as follows: The similarity between the current document to be archived and the candidate documents is calculated using a consistency metric function at four levels: content fingerprint, block-level fingerprint, structural fingerprint, and dependency fingerprint. The archive duplication evaluation value is obtained by multiplying the content similarity, block-level similarity, structural similarity, and dependency similarity together and then taking the fourth power. Deduplication and archiving are then performed.

5. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process of filtering duplicate candidate documents and establishing a version chain entry is as follows: The system compares the archive duplication assessment value with the deduplication threshold in real time. When the archive duplication assessment value is not less than the deduplication threshold, the current document is merged into the main document version chain, written into the version chain incremental record set, and evidence package entries and causal network edge records corresponding to the version are generated. Differential units, references, and permission changes are written into the archive library incremental area without repeatedly writing the complete text to avoid redundancy but ensure that the version chain and causal network remain unbroken. When the archive duplication assessment value is less than the deduplication threshold, the current document is entered into the library as a new main document, and a version chain entry is established. At the same time, the main document evidence package and causal network root node are initialized.

6. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process of generating differential units based on the version chain entry and performing intent strength assessment is as follows: For adjacent versions merged into the same main document version chain, the block-level alignment result and object reference relationship are called. Block-level difference is performed on the text paragraph block and the object list to identify newly added blocks, deleted blocks, replaced blocks, rearranged blocks, difference types, and object reference change types, generating difference units and binding a block-level position index and source version number to each difference unit. Operation intent annotation is constructed based on difference units: continuous difference units are aggregated into editing sessions according to the same author, the same time window, and the same location neighborhood; editing intent labels are output according to difference type within the editing session, and the operation type weight corresponding to the difference unit is obtained through evidence scoring. The change in structural depth is obtained by comparing the absolute value of the change in the hierarchical depth of the document outline before and after the editing session; the maximum structural depth is obtained by analyzing the title hierarchy structure tree of the document version before editing; the number of external references is obtained by counting the number of difference units involving the addition, deletion, or modification of external reference content in this editing session; the number of permission changes is obtained by counting the number of difference units involving changes to the document in this editing session; and the difference unit index is obtained by traversing all difference units contained in this editing session. For each differential unit within an editing session, the operation type weight of the differential unit is multiplied by the structural depth influence coefficient to obtain the basic contribution value. The structural depth change is incremented by one and then logarithmized. The logarithm is then divided by the maximum structural depth of the document, incremented by one, and then logarithmized again. One is added to the ratio to obtain the adjustment factor. The basic contribution value is multiplied by the adjustment factor and then summed sequentially to obtain the total contribution score. The citation change coefficient is multiplied by the number of external citations to obtain the citation bonus, and the permission change coefficient is multiplied by the number of permission changes to obtain the permission bonus. The total contribution score is added to the citation bonus and the permission bonus to obtain the intent intensity assessment value.

7. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process of storing differential cell data and constructing an index strategy is as follows: The strategy for storing and indexing differential cell data is as follows: For editing sessions where the intent strength assessment value is greater than the intent threshold, the original text of the content block and the snapshot of the embedded object are fully preserved in the differential cell; for editing sessions where the intent strength assessment value is less than or equal to the intent threshold, only the content fingerprint is preserved to optimize storage. The intent intensity assessment value is written as a dynamic weight factor into the document field weights of the inverted index.

8. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process of constructing the version causal relationship network and generating the archived evidence package is as follows: A causal traceability graph is constructed based on version chain entry points, differential units, and operation intent annotations: versions are used as nodes, and differential units and reference changes are used as edge relationships; the author, timestamp, source system, session identifier, and editing intent tag are recorded on the edges to form a version causal relationship network; the edges point from the previous version to the next version, and the differential unit is used as an edge attribute; reference changes are used as another type of edge, forming a causal network structure of the version chain. An archived evidence package is generated for each version. The evidence package supports location and verification by version number, author, time window, and reference object. During the verification process, the fingerprint in the evidence package will be compared with the fingerprint tree in the historical archive to verify the validity, completeness, and consistency of the document.

9. The cloud document deduplication, archiving, and traceability management method for office knowledge bases according to claim 1, characterized in that: The specific process of performing consistency verification and outputting the deduplicated archiving results of cloud documents is as follows: The system performs a comprehensive check on the completeness and consistency of statistical evidence elements. It counts and compares the existing text snapshot, differential summary, object list, citation relationships, permission snapshot, and session summary elements in the evidence package with the required evidence element list for the version. The size of the intersection is used as the number of existing elements, and the size of the required element set is used as the number of required elements. The element completeness is calculated by dividing the number of existing elements by the number of required elements, and the type, corresponding index key, and reason code of missing elements are written into the version verification record. Simultaneously, consistency checks are performed on the text fingerprint, object fingerprint, and citation fingerprint, comparing them with the corresponding fingerprints stored in the evidence package and the fingerprint indexes registered in the historical archive. If all three types of fingerprints are identical, the consistency check is considered successful. If any fingerprint is inconsistent, the consistency check fails, and failure handling is performed: the version is marked as needing reconstruction, the archive validity flag is frozen, the inconsistent fingerprint type and source of difference are recorded, and the process entry point for supplementary collection, re-parsing, or rollback and rewriting of the evidence package is triggered. The consistency check conclusion and the cloud document deduplication archive result are output.

10. A cloud-based document deduplication, archiving, and traceability management system for office knowledge bases, employing the cloud-based document deduplication, archiving, and traceability management method for office knowledge bases as described in any one of claims 1-9, characterized in that... include: The multi-source document acquisition module is used to perform event-driven acquisition of document objects, obtain the raw dataset and perform structured parsing to obtain the full data stream, and perform normalization and anomaly marking on the full data stream; The multi-layer fingerprint generation module is used to generate multi-layer fingerprints based on the full data stream, perform candidate duplication retrieval and alignment, evaluate the archive duplication rate, filter duplicate candidate documents, and establish a version chain entry point. The differential redundancy module is used to generate differential units based on the version chain entry point, perform intent strength assessment, store differential unit data, and build an indexing strategy. The knowledge base retrieval service module is used to construct a version causal relationship network, generate archived evidence packages, perform consistency verification, and output the deduplicated archived results of cloud documents.

Images